Multi-view displays and associated systems and methods

ABSTRACT

Techniques for controlling optical behavior of a multi-view display apparatus comprising a first layer comprising first optical elements and a second layer comprising second optical elements. The techniques include obtaining a plurality of scene views; obtaining information specifying a model of the multi-view display apparatus; obtaining information specifying at least one blurring transformation; and generating actuation signals for controlling the multi-view display apparatus to concurrently display a plurality of display views corresponding to the plurality of scene views, the actuation signals comprising first actuation signals for controlling the first optical elements and second actuation signals for controlling the second optical elements, the generating comprising: generating the first actuation signals and the second actuation signals based, at least in part, on the plurality of scene views, the information specifying the model of the multi-view display apparatus, and the information specifying the at least one blurring transformation.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims priority under 35U.S.C. § 120 to U.S. patent application Ser. No. 16/049,423, filed Jul.30, 2018, titled “MULTI-VIEW DISPLAYS AND ASSOCIATED SYSTEMS ANDMETHODS”, which is a continuation of and claims priority under 35 U.S.C.§ 120 to U.S. patent application Ser. No. 15/267,874, filed Sep. 16,2016, titled “MULTI-VIEW DISPLAYS AND ASSOCIATED SYSTEMS AND METHODS”,which claims the benefit under 35 U.S.C. § 119 of U.S. ProvisionalPatent Application No. 62/219,767, filed on Sep. 17, 2015, titled“TECHNIQUES FOR OPTIMIZED DISPLAYS,” and of U.S. Provisional PatentApplication No. 62/245,620, filed on Oct. 23, 2015, titled “ON OPTIMIZEDDISPLAYS,” and of U.S. Provisional Patent Application No. 62/339,830,filed on May 21, 2016, titled “PRINTED LIGHT FIELD DISPLAYS ANDASSOCIATED SYSTEMS AND METHODS,” each of which is hereby incorporated byreference in its entirety.

BACKGROUND

Displays that are capable of creating the illusion of depth have longfascinated viewers. While a conventional two-dimensional display showsobjects that appear at the physical distance of the display, athree-dimensional (3D) display can create visual effects that appear toextend beyond the display itself, both in front of and behind thephysical location of the screen.

One category of 3D displays is “glasses-based” 3D displays, whichrequire a viewer to wear special-purpose eyewear (e.g., 3D glasses) inorder to provide the viewer with a sense of depth. The special-purposeeyewear mediates the light arriving from a more distant display or isable to form an image itself. The eyewear provides stereo pairs ofimages to a viewer's eyes, which in turn provides the viewer with anillusion of depth.

Another category of 3D displays are “glasses-free” 3D displays, whichcan create the illusion of depth without requiring that a viewer of a 3Ddisplay wear special-purpose eyewear or other hardware while viewing the3D display. A glasses-free 3D display may project multiple views of ascene into space in front of the 3D display in one or multipledirections. A glasses-free 3D display may simultaneously displaymultiple views of a scene (e.g., 2 views, tens of views, hundreds ofviews, etc.) to increase the range of viewable locations, increaseperceived display quality, and/or allow a viewer to look “around”displayed objects. Examples of glasses-free 3D displays include parallaxbarrier displays that have a fixed barrier pattern on one layer andsub-images or integral images on another layer, lenticular displays thathave an arrangement of cylindrical lenses on one layer and sub-images orintegral images on another layer, and computational displays thatgenerate content-dependent patterns to display using two or more layersin order to display a 3D scene.

A multi-view 3D display may be capable of simultaneously showingmultiple (two or more) images corresponding to respective multiple viewsat corresponding viewing locations. A viewer may see differentperspectives of a scene from each of the viewing locations. Aglasses-free multi-view 3D display is called an automultiscopic 3Ddisplay. An automultiscopic display may allow a viewer to see aroundvirtual objects as the viewer's viewpoint to the scene changes. As aviewer's head moves from one side of an automultiscopic 3D display toanother, the viewer's eyes may travel through the regions where variousimages are projected from the automultiscopic 3D display. The imagesgenerated by an automultiscopic 3D display may represent variousperspectives of a virtual scene, and through these various perspectivesthe viewer may observe the virtual scene with full motion parallax andstereoscopic depth. An automultiscopic 3D display may generate multipleviews (the particular view seen by a viewer depending on position of theviewer relative to the 3D display), may exhibit binocular disparity,and/or may exhibit motion parallax in both horizontal and verticaldirections.

SUMMARY

Some embodiments provide for a system for generating actuation signalsto control optical behavior of a multi-view display apparatus, themulti-view display apparatus comprising at least two different layersincluding a first layer comprising a first plurality of optical elementsand a second layer comprising a second plurality of optical elements.The system comprises: at least one processor; at least onenon-transitory computer-readable storage medium storingprocessor-executable instructions that, when executed by the at leastone processor, cause the at least one processor to perform: obtaining aplurality of scene views; information specifying a model of themulti-view display apparatus; obtaining information specifying at leastone blurring transformation; and generating a plurality of actuationsignals for controlling the multi-view display apparatus to concurrentlydisplay a plurality of display views corresponding to the plurality ofscene views, the plurality of actuation signals comprising a firstplurality of actuation signals for controlling the first plurality ofoptical elements and a second plurality of actuation signals forcontrolling the second plurality of optical elements, the generatingcomprising: generating the first plurality of actuation signals and thesecond plurality of actuation signals based, at least in part, on theplurality of scene views, the information specifying the model of themulti-view display apparatus, and the information specifying the atleast one blurring transformation.

Some embodiments provide for a method for generating actuation signalsto control optical behavior of a multi-view display apparatus, themulti-view display apparatus comprising at least two different layersincluding a first layer comprising a first plurality of optical elementsand a second layer comprising a second plurality of optical elements.The method comprises using at least one processor configured to perform:obtaining a plurality of scene views; information specifying a model ofthe multi-view display apparatus; obtaining information specifying atleast one blurring transformation; and generating a plurality ofactuation signals for controlling the multi-view display apparatus toconcurrently display a plurality of display views corresponding to theplurality of scene views, the plurality of actuation signals comprisinga first plurality of actuation signals for controlling the firstplurality of optical elements and a second plurality of actuationsignals for controlling the second plurality of optical elements, thegenerating comprising: generating the first plurality of actuationsignals and the second plurality of actuation signals based, at least inpart, on the plurality of scene views, the information specifying themodel of the multi-view display apparatus, and information specifyingthe at least one blurring transformation.

Some embodiments provide for at least one non-transitorycomputer-readable storage medium storing processor-executableinstructions that, when executed by at least one processor, cause the atleast one processor to perform a method for generating actuation signalsto control optical behavior of a multi-view display apparatus, themulti-view display apparatus comprising at least two different layersincluding a first layer comprising a first plurality of optical elementsand a second layer comprising a second plurality of optical elements.The method comprises: obtaining a plurality of scene views; obtaininginformation specifying a model of the multi-view display apparatus;obtaining information specifying at least one blurring transformation;and generating a plurality of actuation signals for controlling themulti-view display apparatus to concurrently display a plurality ofdisplay views corresponding to the plurality of scene views, theplurality of actuation signals comprises a first plurality of actuationsignals for controlling the first plurality of optical elements and asecond plurality of actuation signals for controlling the secondplurality of optical elements, the generating comprising: generating thefirst plurality of actuation signals and the second plurality ofactuation signals based, at least in part, on the plurality of sceneviews, the information specifying the model of the multi-view displayapparatus, and information specifying the at least one blurringtransformation.

Some embodiments provide for a system for generating actuation signalsto control optical behavior of a multi-view display apparatus, themulti-view display apparatus comprising at least two different layersincluding a first layer comprising a first plurality of optical elementsand a second layer comprising a second plurality of optical elements.The system comprises: at least one processor; at least onenon-transitory computer-readable storage medium storingprocessor-executable instructions that, when executed by the at leastone processor, cause the at least one processor to perform: obtaining aplurality of scene views; obtaining information specifying a model ofthe multi-view display apparatus; and generating, based at least in parton the plurality of scene views and the information specifying the modelof the multi-view display apparatus, a plurality of actuation signalsfor controlling the multi-view display apparatus to concurrently displaya plurality of display views corresponding to the plurality of sceneviews, the plurality of actuation signals comprising a first pluralityof actuation signals for controlling the first plurality of opticalelements and a second plurality of actuation signals for controlling thesecond plurality of optical elements, wherein at least two of theplurality of actuation signals each has a Sobel-based high-frequencycontent measure that is greater in value than 0.2 in at least one coloror intensity channel, and wherein the plurality of actuation signals isupdated at a rate of less than 120 Hz (e.g., no greater than 60 Hz).

Some embodiments provide for a system for generating signals to controloptical behavior of a multi-view display apparatus, the multi-viewdisplay apparatus comprising at least two different layers including afirst layer comprising a first plurality of optical elements and asecond layer comprising a second plurality of optical elements. Thesystem comprises: at least one processor; at least one non-transitorycomputer-readable storage medium storing processor-executableinstructions that, when executed by the at least one processor, causethe at least one processor to perform: obtaining a plurality of sceneviews; information specifying a model of the multi-view displayapparatus; obtaining information specifying at least one blurringtransformation; and generating a plurality of signals for controllingthe multi-view display apparatus to concurrently display a plurality ofdisplay views corresponding to the plurality of scene views, theplurality of signals comprising a first plurality of signals forcontrolling the first plurality of optical elements and a secondplurality of signals for controlling the second plurality of opticalelements, the generating comprising: generating the first plurality ofsignals and the second plurality of signals based, at least in part, onthe plurality of scene views, the information specifying the model ofthe multi-view display apparatus, and the information specifying the atleast one blurring transformation. A signal may comprise an actuationsignal such that the first plurality of signals may comprise a firstplurality of actuation signals and the second plurality of signals maycomprise a second plurality of actuation signals.

Some embodiments provide for a multi-view display apparatus comprising:a first layer comprising a first plurality of optical elements; a secondlayer comprising a second plurality of optical elements, the secondlayer being separated from the first layer by a distance; and controlcircuitry comprising: first circuitry configured to control the firstplurality of optical elements; and second circuitry configured tocontrol the second plurality of optical elements, wherein the controlcircuitry is configured to: receive a first plurality of actuationsignals and a second plurality of actuation signals generated, based atleast in part on a plurality of scene views, information specifying amodel of the multi-view display apparatus and information specifying atleast one blurring transformation, control the multi-view displayapparatus to concurrently display a plurality of views corresponding tothe plurality of scene views at least in part by: controlling the firstplurality of optical elements to display first content using the firstplurality of actuation signals, and controlling the second plurality ofoptical elements to display second content using the second plurality ofactuation signals.

Some embodiments provide for a method of manufacturing a light fieldprint, the light field print comprising at least two differenttransparent layers including a front transparent layer and a backtransparent layer. The method comprises: obtaining content to berendered using the light field print, the content comprising a pluralityof scene views; obtaining printing process information; generating,based at least in part on the content and the printing processinformation, a first target pattern for the front transparent layer anda second target pattern for the back transparent layer; printing thefirst target pattern on the front transparent layer by depositingprinting material on the front transparent layer in accordance with thefirst target pattern; and printing the second target pattern on the backtransparent layer by depositing printing material on the backtransparent layer in accordance with the second target pattern, whereinthe front transparent layer is spaced in depth at a distance from theback transparent layer, which distance is less than or equal to agreater of six millimeters and L/60, wherein L is a maximum linearextent of a larger one of the front transparent layer and the backtransparent layer, when the front transparent layer and the backtransparent layer are different sizes, and a maximum linear extent ofthe front transparent layer when the front transparent layer and theback transparent layer are a same size.

The foregoing is a non-limiting summary of the invention, which isdefined by the attached claims.

BRIEF DESCRIPTION OF DRAWINGS

Various aspects and embodiments will be described with reference to thefollowing figures. It should be appreciated that the figures are notnecessarily drawn to scale.

FIG. 1A shows an illustrative system for generating actuation signalsfor controlling a multi-view display and controlling the multi-viewdisplay using the generated actuation signals, in accordance with someembodiments of the technology described herein.

FIG. 1B shows an illustrative system for generating patterns to beprinted on layers of a light field print and printing the generatedpatterns on the layers of the light field print, in accordance with someembodiments of the technology described herein.

FIG. 2 is an illustrative block diagram of the processing performed togenerate actuation signals for controlling a multi-view display, inaccordance with some embodiments of the technology described herein.

FIG. 3 shows an example optimization problem that may be solved as partof generating actuation signals for controlling a multi-view displayand/or as part of generating patterns for printing on one or more layersof a light field print, in accordance with some embodiments of thetechnology described herein.

FIG. 4 illustrates aspects of a gradient descent technique forgenerating one or more solutions to the optimization problem shown inFIG. 3, in accordance with some embodiments of the technology describedherein.

FIG. 5 illustrates an example of an update rule that may be used forgenerating one or more solutions to the optimization problem shown inFIG. 3, in accordance with some embodiments of the technology describedherein.

FIG. 6 shows another example of an optimization problem that may besolved as part of generating actuation signals for controlling amulti-view display and/or as part of generating patterns for printing onone or more layers of a light field print, in accordance with someembodiments of the technology described herein.

FIG. 7 illustrates aspects of a gradient descent technique forgenerating one or more solutions to the optimization problem shown inFIG. 6, in accordance with some embodiments of the technology describedherein.

FIG. 8 shows another example of an optimization problem that may besolved as part of generating actuation signals for controlling amulti-view display and/or as part of generating patterns for printing onone or more layers of a light field print, in accordance with someembodiments of the technology described herein.

FIG. 9 illustrates aspects of a gradient descent technique forgenerating one or more solutions to the optimization problem shown inFIG. 8, in accordance with some embodiments of the technology describedherein.

FIG. 10 illustrates aspects of another technique that may be used togenerate one or more solutions to the optimization problem shown in FIG.8, in accordance with some embodiments of the technology describedherein.

FIG. 11 illustrates aspects of a technique that may be used to generateone or more solutions to the optimization problem shown in FIG. 8 inwhich a multiplicative update rule enforcing non-negativity of theactuation signals is employed, in accordance with some embodiments ofthe technology described herein.

FIG. 12 illustrates aspects of another technique that may be used togenerate one or more solutions to the optimization problem shown in FIG.8 in which a multiplicative update rule enforcing non-negativity of theactuation signals is employed, in accordance with some embodiments ofthe technology described herein.

FIG. 13 illustrates a general form of the multiplicative update ruleshown in FIGS. 11 and 12, in accordance with some embodiments of thetechnology described herein.

FIG. 14 illustrates simulated views generated by a multi-view display inaccordance with some embodiments of the technology described herein.

FIG. 15 is a flowchart of an illustrative process 1500 for generatingactuation signals to control optical behavior of a multi-view displayapparatus in accordance with some embodiments of the technologydescribed herein.

FIGS. 16A and 16B illustrate pixel orderings in a display systemrespectively with and without a non-linear mapping between pixel indicesand the locations of the associated output light ray intensities, inaccordance with some embodiments of the technology described herein.

FIG. 17 illustrates a view cone for a viewer observing a multi-viewdisplay, in accordance with some embodiments of the technology describedherein.

FIGS. 18A and 18B show detail and far-field views of a pinhole barrierdisplay.

FIG. 18C illustrates a system configured for use with non-negativematrix factorization methods.

FIGS. 19A and 19B illustrate color filter responses for use in a colorfilter array in a multi-view display, in accordance with someembodiments of the technology described herein.

FIG. 20 illustrates techniques for compensating for internal reflectionswithin a multi-view display, in accordance with some embodiments of thetechnology described herein.

FIG. 21 illustrates a multi-view display comprising diffusers, inaccordance with some embodiments of the technology described herein.

FIGS. 22A-F illustrate aspects of reflection mode multi-view displays,in accordance with some embodiments of the technology described herein.

FIG. 23 illustrates a multi-layer light field display, in accordancewith some embodiments of the technology described herein.

FIGS. 24A and 24B illustrates embodiments of multi-view displays thatmay be used for augmented reality and other applications that make useof visual accommodation effects, in accordance with some embodiments ofthe technology described herein.

FIG. 25 shows an illustrative example of using a mobile device forcalibrating a multi-view display, in accordance with some embodiments ofthe technology described herein.

FIG. 26 shows another illustrative system for generating patterns to beprinted on layers of a light field print and printing the generatedpatterns on the layers of the light field print, in accordance with someembodiments of the technology described herein.

FIGS. 27A and 27B show illustrative examples of a light field print,manufactured in accordance with some embodiments of the technologydescribed herein.

FIG. 28 shows another illustrative example of a light field print,manufactured in accordance with some embodiments of the technologydescribed herein.

FIG. 29 shows an illustrative example of a light field printmanufactured using a self-aligned printing method, in accordance withsome embodiments of the technology described herein.

FIG. 30 shows an illustrative system for adaptively aligning theprinting process used for printing a layer of a light field print, inaccordance with some embodiments of the technology described herein.

FIG. 31 illustrates an example of a print service, in accordance withsome embodiments of the technology described herein.

FIGS. 32 and 33 show an example light field print for use with a glasssurface, such as a window, in accordance with some embodiments of thetechnology described herein.

FIG. 34 is a flowchart of an illustrative process 3400 for manufacturinga light field print, in accordance with some embodiments of thetechnology described herein.

FIG. 35 shows, schematically, an illustrative computer 3500 on which anyaspect of the technology described herein may be implemented.

DETAILED DESCRIPTION

The inventors have recognized and appreciated that conventionalautomultiscopic 3D displays may be improved upon. Conventionalautomultiscopic 3D displays do not allow for both high spatialresolution and high angular resolution—the manufacturer must trade-offthese two display characteristics even though both of them are desirableto consumers. On the one hand, images displayed by an automultiscopic 3Ddisplay will appear blurry or jagged when the automultiscopic 3D displaydoes not have sufficient spatial resolution. On the other hand, when anautomultiscopic 3D display does not have sufficient angular resolution,the 3D effect of regions that appear to pop in or out of anautomultiscopic display will degrade quickly the further the imageappears to float from the physical plane of the display. For example,images displayed by an automultiscopic 3D display having insufficientangular resolution may appear increasingly blurry with distance from thephysical plane of the 3D display.

The inventors have recognized and appreciated that conventionalautomultiscopic 3D displays do not allow for dynamically trading offspatial and angular resolution in order to efficiently use the availableresolution of a 3D display by dynamically matching the demands of ascene to be displayed. For example, a single scene may have regions thatrequire high spatial resolution (e.g., one or more regions that has arapidly varying pattern) and regions that require high angularresolution (e.g., one or more regions that appears to pop-out far fromthe physical plane of the display). However, for a conventionalautomultiscopic 3D display, the trade-off between spatial and angularresolution of the display must be selected when the 3D display is beingmanufactured, and it must remain constant across the entire displaysurface (i.e., it cannot be dynamically adjusted based on the nature ofthe content to be displayed by the 3D display). As a result, aconventional automultiscopic 3D display can achieve high angularresolution (e.g., for displaying large pop-out effects) only by tradingaway spatial resolution, which will make all images more blurry,including in the regions that do not appear to pop out of the displayand, therefore, do not require high angular resolution.

The inventors have recognized and appreciated that another shortcomingof conventional automultiscopic 3D displays is that their manufactureoften requires using optical elements that can be costly to manufactureand challenging to calibrate. Current technology available formanufacturing automultiscopic 3D displays does not provide asufficiently high spatio-angular resolution relative to their high costand difficulty of manufacture. Although attempts were made to usecomputational displays to address some of these problems, the resultingdisplays are thicker than conventional automultiscopic displays, sufferfrom narrow viewing angles, are challenging to manufacture, and areoptically inefficient with conventional display hardware.

The inventors have developed a new class of automultiscopic 3D displaysthat allow for dynamically trading off spatial and angular resolution tomatch the demands of a scene to be displayed. The new automultiscopic 3Ddisplays are computational 3D displays comprising multiple layerscontrolled by content-dependent actuation signals to display a 3D scene.Using computation to dynamically achieve a desired balance betweenspatial and angular resolution in an automultiscopic 3D display providesthe ability to have sharp in-plane text and graphics and also have largedegrees of perceived pop-out in the same display and the same scene,which is something that is not possible with conventionalautomultiscopic 3D displays.

The inventors have recognized and appreciated that considering thecapabilities of the human visual system to resolve individual points onthe surface of an automultiscopic 3D display provides additional degreesof freedom in designing the automultiscopic 3D display. Such additionaldegrees of freedom may be used to dynamically tradeoff spatialresolution and angular resolution and to represent 3D scenes withgreater depth, while shrinking the thickness of the automultiscopic 3Ddisplay itself.

Accordingly, in some embodiments, the actuation signals used forcontrolling automultiscopic 3D displays are generated at least in partby using one or more blurring transformations that may be designedand/or selected based on perceptual capabilities of the human visualsystem. For example, in some embodiments, by shifting reconstructionerror in each of the views generated by an automultiscopic 3D displayoutside the bandwidth of the human vision system, crosstalk between theviews can be significantly reduced, resulting in improved performancefrom the perspective of the viewer. The perceived bandlimited behaviormay arise as a result of various factors, including but not limited tothe finite resolution of the human retina, focus blur, higher-orderoptical effects, and diffractive effects in the display hardware orhuman vision system. Although, perceptually-inspired weighting has beenused in some multi-layer displays as a weighting constraint onindividual rays, the blurring transformations used in some of theembodiments described herein may impose a bandwidth constraint on theensemble of rays in each view of the display, which gives rise to a newoptimization problem that the techniques described herein may be used tosolve.

The inventors have also recognized and appreciated that using one ormore blurring transformations designed and/or selected based onperceptual capabilities of the human visual system to dynamicallygenerate signals for controlling multiple layers of an automultiscopic3D display may also result in a significant brightness increase ascompared to conventional parallax barrier based automultiscopic 3Ddisplays. For example, a conventional parallax barrier basedautomultiscopic 3D display producing a total of N views would result in1/N factor in overall brightness, potentially reducing brightnesssubstantially. By contrast, some embodiments provide for automultiscopic3D displays having a much higher overall brightness for the same numberof views. Indeed, any increase in brightness over a conventionalparallax-based technique for an equivalent number of generated views, incombination with the utilization of one or more blurringtransformations, may be indicative of the utilization of the techniquesdeveloped by the inventors.

The inventors have also recognized and appreciated that by adjustingbasic class of the underlying optical modulators, automultiscopic 3Ddisplays can be made less costly, and can be made to perform better interms of optical efficiency. In comparison to lenticular printeddisplays, one important contribution by the inventors is to create athin and light efficient method to create glasses-free 3D printeddisplays that does not require any refractive optical element.

Some embodiments of the technology described herein address some of theabove discussed drawbacks of conventional automultiscopic 3D displays.However, not every embodiment addresses every one of these drawbacks,and some embodiments may not address any of them. As such, it should beappreciated that aspects of the technology described herein are notlimited to addressing all or any of the above discussed drawbacks ofconventional automultiscopic 3D displays.

Accordingly, some embodiments provide for a novel class ofautomultiscopic multi-view 3D displays, developed by the inventors, andtechniques for controlling such displays to generate desired sceneviews. The automultiscopic 3D displays developed by the inventors arecomputational displays in that they are controlled by actuation signalsdynamically determined using the content to be displayed. In someembodiments, the actuation signals may be determined based, at least inpart, using one or more blurring transformations. The blurringtransformations may be designed and/or selected based on characteristicsof the human visual system. Non-limiting examples of blurringtransformations are provided herein.

In describing blurring transformations, we generally refer to thelimited spatial, temporal, and spatiotemporal bandwidth associated withtwo-dimensional static and moving images corresponding to individualviews of an optimized multi-layer display. This is different from and isin contrast to limiting bandwidth in ray space. Qualitatively, limitingbandwidth in ray space results in increased blurring in a particularview, whereas the techniques described herein decreases effectiveblurring in a particular view. The described techniques achieve thisgoal by recognizing that some band-limited blurring naturally occurs ineach view due to perceptual effects, and this allows for additionaldegrees of freedom, which can be used in reducing blurring due tointer-view crosstalk.

I. Controlling Optical Behavior of a Multi-View Display Using One orMore Blurring Transformations

Some embodiments provide for techniques for generating actuation signalsto control optical behavior of a multi-view display apparatus includinga first layer comprising first optical elements and a second layercomprising second optical elements. In some embodiments, the techniquesinclude: (1) obtaining scene views (e.g., obtaining a set of scene viewscorresponding to a respective set of positions of one or more viewers ofthe multi-view display apparatus, for example, relative to the displayapparatus); (2) obtaining information specifying a model of themulti-view display apparatus; (3) obtaining information specifying atleast one blurring transformation (e.g., obtaining informationspecifying a blurring transformation for each of the scene views); (4)generating actuation signals for controlling the multi-view displayapparatus to concurrently generate display views corresponding to thescene views, the actuation signals comprising first actuation signalsfor controlling the first optical elements and second actuation signalsfor controlling the second optical elements; and (5) controlling themulti-view display apparatus using the generated actuation signals(e.g., by providing the first and second actuation signals to circuitryfor controlling the multi-view display apparatus and using the circuitryto control the first optical elements using the first actuation signalsand the second optical elements using the second actuation signals).

In some embodiments, generating the actuation signals used forcontrolling the multi-view display apparatus may include generating thefirst actuation signals and the second actuation signals based, at leastin part, on the scene views, the information specifying a model of themulti-view display apparatus, and the information specifying the atleast one blurring transformation.

In some embodiments, the actuation signals may be generated using aniterative optimization technique. In some embodiments, generating theactuation signals includes: (1) generating an initial set of actuationsignals; (2) iteratively updating the initial set of actuation signalsto produce a sequence of intermediate sets of actuation signals; and (3)and outputting a last set of actuation signals in the sequence ofintermediate sets of actuation signals as the actuation signals to usefor controlling the optical behavior of the multi-view displayapparatus.

In some embodiments, iterative updating the initial set of actuationsignals may be performed based, at least in part, on the scene views,the information specifying the model of the multi-view display, andinformation specifying the at least one band-limiting transformation.Iteratively updating the first set of actuation signals may include: (1)determining, using the information specifying the model of themulti-view display apparatus and the first set of actuation signals, afirst set of display views corresponding to display views that would begenerated by the multi-view display apparatus if the first set ofactuation signals were used to control the multi-view display apparatus;(2) determining, using the at least one blurring transformation, ameasure of error between the first set of display views and theplurality of scene views; and (3) updating the first set of actuationsignals based on the measure of error between the first set of displayviews and the plurality of scene views. In some embodiments, theupdating may be performed multiplicatively and subject to non-negativityconstraints on the actuation signals.

In some embodiments, the multi-view display apparatus may generategrayscale content on one layer and color content on another layer.Accordingly, in some embodiments, first and second actuation signals aregenerated such that, when the first actuation signals are used tocontrol the first layer, the first layer displays color content and,when the second actuation signals are used to control the second layer,the second layer displays grayscale content.

As described herein, in some embodiments, one or more blurringtransformations may be used to generate actuation signals forcontrolling optical behavior of a multi-view 3D display. In someembodiments, for example, such blurring transformations may be appliedto one or more scene views and/or display views when iterativelyidentifying the actuation signals to use for driving the multi-view 3Ddisplay. However, blurring transformations may be used in any othersuitable way when generating actuation signals for controlling opticalbehavior of a multi-view 3D display, as the utilization of blurringtransformations is not limited to the application of suchtransformations to scene views and/or display views (e.g., in someembodiments, blurring transformations may be applied to error views, asdescribed in greater detail below).

In such embodiments, applying a blurring transformation to an image(e.g., a scene view or any other suitable image) may include convolvingthe image with the band-limiting transformation in the spatial domain ormultiplying the 2D Fourier transform (or other frequency transform) ofthe band-limiting transformation with a corresponding transformation ofthe image.

In some embodiments, a blurring transformation may comprise aband-limiting function. The band-limiting function may be a 2D function.In some embodiments, a band-limiting function may have a 2D Fouriertransform whose magnitude, on average or asymptotically, may decreasewith increasing spatial frequency. For example, one illustrative classof image transformations takes the following form:y[u,v]=Σ_(s=−∞) ^(∞)Σ_(t=−∞) ^(∞) h[u−s,v−t]x[s,t],where the input image is denoted by x, the output image is denoted by y,x[u,v] is the intensity of the input image x evaluated at horizontallocation u and vertical location v, and y[u,v] is the intensity of theoutput image y evaluated at horizontal location u and vertical locationv. Then the h[u,v] may specify parameters of a band-limiting functionfor processing input image x to obtain output image y if it has a 2DFourier transform whose magnitude, on average or asymptotically,decreases with increasing spatial frequency. Illustrative non-limitingexamples of such band-limiting functions include:

y[u, v] = (x[u + 1, v − 1] + x[u + 1, v] + x[u + 1, v + 1] + x[u, v − 1] + x[u, v] + x[u, v + 1] + x[u − 1, v − 1] + x[u − 1, v] + x[u − 1, v + 1])/9;${{y\lbrack {u,v} \rbrack} = \frac{{x\lbrack {{u + 1},v} \rbrack} + {x\lbrack {{u - 1},v} \rbrack} + {x\lbrack {u,{v - 1}} \rbrack} + {x\lbrack {u,{v + 1}} \rbrack} + {x\lbrack {u,v} \rbrack}}{5}};$${{y\lbrack {u,v} \rbrack} = \frac{{x\lbrack {{u + 1},v} \rbrack} + {x\lbrack {{u - 1},v} \rbrack} + {x\lbrack {u,{v - 1}} \rbrack} + {x\lbrack {u,{v + 1}} \rbrack} + {4{x\lbrack {u,v} \rbrack}}}{8}};$y[u, v] = (0.25x[u + 1, v − 1] + 0.5x[u + 1, v] + 0.25x[u + 1, v + 1] + 0.5x[u, v − 1] + x[u, v] + 0.5x[u, v + 1] + 0.25x[u − 1, v − 1] + 0.5x[u − 1, v] + 0.25x[u − 1, v + 1])/4;  andy[u, v] = (0.25x[u + 1, v − 1] + 0.5x[u + 1, v] + 0.25x[u + 1, v + 1] + 0.5x[u, v − 1] + x[u, v] + 0.5x[u, v + 1] + 0.25x[u − 1, v − 1] + 0.5x[u − 1, v] + 0.25x[u − 1, v + 1] + 0.25[u − 1, v + 2])/4.25.

In some embodiments, a blurring transformation may be any linear ornon-linear function that, when applied to an image, reduces the amountof high-frequency content and/or fine detail in the image.

In some embodiments, a blurring transformation may be any function thatapplies a model of the human visual system to an image. For example, ablurring transformation may be any function that applies a model ofhuman visual acuity to an image. As another example, a blurringtransformation may be any function that applies a model of humancontrast sensitivity to an image.

In some embodiments, a blurring transformation may comprise a spatialand/or temporal band-limiting function representing an approximation ofthe band-limited behavior of the human vision system. For example, ablurring transformation may comprise a band-limiting function tailoredto the long term vision characteristics of a specific individual (e.g.,the specific vision deficiencies of the individual). As another example,a blurring transformation may comprise a band-limiting function tailoredto the short-term vision characteristics of an individual viewer (e.g.,taking into account the viewer's specific viewing position orinstantaneous accommodation focal length).

In some embodiments, applying a blurring transformation to an imagecomprises spatially convolving (or performing any equivalent calculationin the spatial or other domain such as, for example, multiplication inthe Fourier domain) the image with another function. For example,applying a blurring transformation to an image may comprise spatiallyconvolving the image with a point spread function of an optical system(e.g., a camera, optics of a human eye, optical effects of sending lightthrough a very small home the size of a pixel). As a specific example,applying a blurring transformation to an image may comprise spatiallyconvolving the image with a kernel representing a shape of an apertureor a frequency-domain representation of the shape of the aperture. Asanother example, applying a blurring transformation to an image maycomprise spatially convolving the image with a two-dimensional,spatially discrete point spread response, for which the sum of theresponse, taken over all discrete entries, is greater than or equal tothe l₂-norm of the response, taken over all discrete entries. As yetanother example, applying a blurring transformation to an image maycomprise spatially convolving the image with a two-dimensional Gaussianfunction.

In some embodiments, applying a blurring transformation to an image maycomprise applying a binary morphological transformation (e.g., anerosion, a dilation, a morphological opening, and a morphologicalclosing) to the image. In some embodiments, applying a blurringtransformation to an image may comprise applying a rank filter (e.g., amedian filter, a majority filter, etc.) to the image.

In some embodiments, a blurring transformation may be specified as acost function in a transformed color space (e.g., utilizing distinctspatio temporal band-limited response characteristics for luminance andchrominance channels) or using other color decompositions.

In some embodiments, a blurring transformation may represent the effectsdue to diffractive interactions between layers of a multi-view displaydevice (or layers of a light field print) and/or effects due to one ormore optical diffusers or other passive layers.

Regardless of the particular form of blurring transformation(s) used togenerate actuation signals for controlling a layers of a multi-view 3Ddisplay, the actuation signals obtained using the blurringtransformations will generally have a significant amount of highfrequency content. To make this notion precise, we introduce a so-calledSobel edge detector, which is an edge detection filter that may be usedto assess the amount of high-frequency content in an image.

Given a two-dimensional image representing a single color or intensitychannel of an actuation signal, denoted x_(k), its Sobel magnitude image{circumflex over (x)}_(k) may be computed by first computing the Sobelgradients G_(k) ^((x)) and G_(k) ^((y)) according to:

${G_{k}^{(x)} = {{\begin{bmatrix}{- 1} & 0 & {+ 1} \\{- 2} & 0 & {+ 2} \\{- 1} & 0 & {+ 1}\end{bmatrix}*x_{k}\mspace{14mu}{and}\mspace{14mu} G_{k}^{(y)}} = {\begin{bmatrix}{- 1} & {- 2} & {- 1} \\0 & 0 & 0 \\{+ 1} & {+ 2} & {+ 1}\end{bmatrix}*x_{k}}}},$where * denotes the 2-dimensional signal processing convolutionoperation, and then computing the Sobel magnitude image according to:

${{\hat{x}}_{k} = {\frac{1}{\sqrt{32}}\sqrt{G_{k}^{{(x)}^{2}} + G_{k}^{{(y)}^{2}}}}},$where the square root function, summation and squaring functions areperformed on a pixel-by-pixel basis. The scale factors used in computingthe Sobel magnitude image {circumflex over (x)} are consistent with theimplementation of the Sobel-based edge detection algorithm appearing inthe open-source graphics package GIMP.

In addition, we introduce a Sobel-based high frequency content measureϕ_(k), defined as the ratio of the average pixel value of the Sobelmagnitude image {circumflex over (x)}_(k) to the average pixel value ofthe corresponding pattern image (actuation signal) x_(k). Accordingly,the Sobel-based high frequency content measure ϕ_(k) may be obtainedaccording to:

${\phi_{k} = \frac{{AV}\;{GPX}\;( {\overset{\hat{}}{x}}_{k} )}{{AV}\;{GPX}\;( x_{k} )}},$where AVGPX({circumflex over (x)}_(k)) denotes the average pixel valueof {circumflex over (x)}_(k) and AVGPX(x_(k)) denotes the average pixelvalue of x_(k). All mathematical operations for computing a Sobelmagnitude image and the Sobel-based high frequency content measure areperformed in continuous value space and independently of whether theactuation signal is binary valued. For multichannel actuation signals,the Sobel magnitude image and high-frequency content measure may beobtained by operating on each channel individually.

As described herein, the actuation signals obtained using the blurringtransformations in accordance with some embodiments of the technologydescribed herein may have a significant amount of high frequencycontent. For example, two or more of the plurality of actuation signalsmay each have a Sobel-based high-frequency content measure that isgreater in value than 0.2 (e.g., between 0.2 and 1.0) in at least onecolor or intensity channel. By contrast, natural image s may have valuesin the range of (0.001-0.06).

It should also be appreciated that aspects of the technology describedherein are not limited to explicitly using one or more blurringtransformations to generate actuation signals for controlling opticalelements in multi-view 3D displays. In some embodiments, actuationsignals may be generated using algorithms that do not explicitly containa blurring transformation, but otherwise generate actuation signalsconsistent with the overall approach (e.g., having at least a thresholdvalue for a Sobel-based high frequency content measure). As one example,in some embodiments, any heuristic technique for shaping the error indisplay images generated by a multi-view 3D display to be out of band ofthe human visual system may be employed.

II. Multi-View Display Arrangements

The techniques for generating actuation signals for controlling amulti-view display apparatus may be used with numerous types ofmulti-view 3D displays described herein. In some embodiments, themulti-view 3D display may be an automultiscopic display. In someembodiments, the multi-view 3D display may be a computational display.

In some embodiments, the multi-view 3D display may be a multi-layerdisplay comprising multiple (e.g., two, three, four, five, etc.) layersof optical elements. A layer in the multi-view 3D display may be apassive optical layer, an active optical layer, or a layer having bothpassive and active elements. Examples of passive optical layers include,but are not limited to, polarizers, diffusers, brightness-enhancingfilms, wave retarders, color filters, holographic layers, parallaxbarriers, and lenslet arrays. Examples of active optical layers include,but are not limited to, single- and multi-layer liquid crystal displayscreens, a layer comprising light emitting diodes (LEDs), fluorescentbacklight, organic LED (OLED) backlight, an OLED layer, a layercomprising electronically focusable lenses, and multilayer polarizationrotators. In some embodiments, a multi-view display apparatus mayinclude a first layer comprising first optical elements, a second layercomprising second optical elements and separated from the first layer bya distance, and control circuitry configured to control the first layerand the second layer. The control circuitry may comprise first circuitryconfigured to control the first optical elements and second circuitryconfigured to control the second optical elements. The control circuitrymay be configured to: (1) receive first actuation signals and secondactuation signals that were generated, based at least in part on sceneviews, information specifying a model of the multi-view displayapparatus and information specifying at least one blurringtransformation; and (2) control the multi-view display apparatus toconcurrently display views corresponding to the scene views at least inpart by: controlling the first optical elements to display first contentusing the first actuation signals, and controlling the second opticalelements to display second content using the second actuation signals.Examples of blurring transformations are provided herein.

In some embodiments, controlling the first plurality of optical elementscomprises controlling the first optical elements using the firstactuation signals to display grayscale content; and controlling thesecond optical elements comprises controlling the second opticalelements using the second actuation signals to display color content. Insome embodiments, controlling the first optical elements comprisescontrolling the first optical elements using the first actuation signalsto display content that is binary in each of intensity or colorchannels; and controlling the second optical elements comprisescontrolling the second optical elements using the second plurality ofactuation signals to display content that is binary in each intensity orcolor channel.

In some embodiments, the first and second layers may both be activelayers. For example, the first and second layers may each include LCDpanels. As another example, the first layer may include an array of LEDsand the second layer may include an LCD panel. In other embodiments, oneof the first and second layers may be an active layer and the otherlayer may be a passive layer. In yet other embodiments, both the firstand second layers may be passive layers. In some embodiments, at leastone of the first and second layers may be reflective and/ortransmissive. In some embodiments, at least one of the first and secondlayers may include a transflective LCD. In some embodiments, at leastone of the first layer and the second layer has a contrast of less than1:100. In some embodiments, the pitch of optical elements in the firstand/or second layers of optical elements may be less than or equal to0.005 inches.

In some embodiments, the first layer may include a first color filterarray and the second layer may include a second color filter array. Eachof the first and second color filter arrays may include color filtershaving at least at threshold full-width halfmax response (e.g., at least50 nm, at least 60n, at least 70 nm, at least 80 nm, at least 90 nm, atleast 100 nm, etc.). In some embodiments, color channels of themulti-view apparatus may be optimized jointly.

In some embodiments, the first layer may be spaced in depth at adistance of less than six millimeters from the second layer. In someembodiments, the first layer may be spaced in depth at a distance fromthe second layer that is no more than the greater of the following twoquantities: six millimeters, and 1/60th of the maximum linear extent ofthe larger of the first layer and the second layer.

In some embodiments, the multi-view display apparatus may include one ormore layers and/or components in addition to the first and secondlayers. For example, in some embodiments, the multi-view displayapparatus may include one or more diffusers (e.g., a diffuser placedbetween the first and second layers). As another example, in someembodiments, the multi-view display apparatus may include a backlightunit. In some embodiments, at least 90% of the light emitted by thebacklight unit may be emitted over an angular region containing expectedviewing locations (by one or more viewers) of the multi-view apparatus.Additionally or alternatively, the multi-view apparatus may include oneor more vertically-oriented diagonally-oriented, orhorizontally-oriented lens sheets, one or more lenslet arrays,angle-expanding film, light concentrating film, one or more polarizers,one or more diffractive elements, one or more holographic elements, oneor more optical diffusers, one or more reflective elements includingspecular and diffuse reflective elements, one or more optical films, oneor more wave retarders (e.g., ½ wave plates).

In some embodiments, the multi-view display apparatus may be designed tobe viewed from a distance of no more than one foot from an eye of theviewer. Some embodiments provide for a fixture comprising the multi-viewdisplay apparatus that positions the multi-view display apparatus at adistance of less than six inches from an eye of the viewer. For example,the multi-view display apparatus may be part of a wearable (e.g.,virtual reality) headset worn by a viewer.

III. Techniques for Manufacturing Light Field Prints

The inventors have developed techniques of printing on transparent mediafor the purpose of presenting 3D information to viewers. The resultingprints are layered passive 3D display arrangements, having multiplepassive layers, and are referred to as “light field prints” herein.Described herein are techniques for rapid, robust, and precisemanufacturing of light field prints.

The inventors have recognized and appreciated that the process ofcreating printed patterns intended for light field rendition is moredemanding than that of creating printed patterns for conventional 2Dprinting. In light field printing, for example, features well below thevisual acuity of the human eye may create effects that alter the visibleperformance of a multi-layer light field print. Recognizing this fact,it is necessary to develop techniques to improve the performance ofprinting techniques at all levels of the technology stack, from thesoftware representation of the patterns to be printed, to the physicalmethods of printing, to the alignment and calibration of the printer andthe printed results.

It should be appreciated that generating a glasses-free 3D light fieldprint is entirely different “3D printing.” In 3D printing, physicalstructures of a desired shape are produced directly, for example, byadditive manufacturing (e.g., sequentially depositing layers of meltedmaterials to build up the desired structure). In glasses-free 3D lightfield printing, two or more flat printed layers are produced and stackedon top of one another, such that when observed from a range of anglesthe viewer perceives a physical object to be floating in the vicinity ofthe printed layers. The physical extent of the layers is generally muchsmaller than that of the perceived object. By way of example, if a 5cm×5 cm×5 cm cube were to be 3D printed, it would require specializedhardware capable of depositing physical material in a volume, and theobject would occupy a volume of 5 cm×5 cm×5 cm upon completion of theprint. On the other hand, a light field print of the same 5 cm×5 cm×5 cmcube would require a printer substantially similar to a standard officeprinter to print patterns on two 0.1 mm thick sheets, which whenseparated by 0.8 mm will produce a virtual image of the same cube, suchthat the total physical volume of the print is 5 cm×5 cm×1 cm.

Some embodiments provide for a method of manufacturing a light fieldprint comprising at least two different transparent layers including afront transparent layer and a back transparent layer. The methodincludes: (1) obtaining content to be rendered using the light fieldprint, the content comprising a plurality of scene views; (2) obtainingprinting process information; (3) generating, based at least in part onthe content and the printing process information, a first target patternfor the front transparent layer and a second target pattern for the backtransparent layer; (4) printing the first target pattern on the fronttransparent layer by depositing printing material (e.g., ink or toner)on the front transparent layer in accordance with the first targetpattern at a desired dot pitch (e.g., less than 0.0025 inches); (5)printing the second target pattern on the back transparent layer bydepositing printing material (e.g., ink or toner) on the backtransparent layer in accordance with the second target pattern at adesired dot pitch (e.g., less than 0.0025 inches); and (6) assembling(e.g., using adhesives in some embodiments) the light field print fromthe front transparent layer and the back transparent layer such that thefront transparent layer is spaced in depth at a distance from the backtransparent layer. This distance may be less than or equal to a greaterof six millimeters and L/60, wherein L is a maximum linear extent of alarger one of the front transparent layer and the back transparentlayer, when the front transparent layer and the back transparent layerare different sizes, and a maximum linear extent of the fronttransparent layer when the front transparent layer and the backtransparent layer are a same size.

In some embodiments, the method for manufacturing the light field alsoincludes obtaining information specifying at least one blurringtransformation (examples of which are provided herein) and generatingthe first and second target patterns by using the information specifyingthe at least one blurring transformation.

In some embodiments, generating the first target pattern may beperformed by: (1) generating, based at least in part on the content andthe printing process information, an initial first target pattern forthe front printed layer and an initial second target pattern for theback printed layer; (2) modifying the initial first target pattern tocompensate for effects of print and/or medium dynamics to obtain thefirst target pattern; and (3) modifying the initial second targetpattern to compensate for effects of print and/or medium dynamics toobtain the second target pattern.

In some embodiments, compensating a target pattern for print and/ormedium dynamics may include compensating the target pattern for effectsof dot gain, for example, by applying spatial linear filtering to thetarget pattern or in any other suitable way. In some embodiments,compensating a target pattern for print and/or medium dynamics mayinclude compensating the target pattern for effects of printing materialbleed and/or maximum allowable printing material density of the fronttransparent layer, for example, by eliminating pixels in the targetpattern so that printing material is not deposited on the fronttransparent layer at locations of the eliminated pixels or in any othersuitable way.

In some embodiments, assembling the light field print comprises firstprinting the second target pattern on the back transparent layer, thenplacing the front transparent layer on the back transparent layer beforeprinting the first target pattern on the front transparent layer, andthen printing the first target pattern on the front transparent layer.

It should be appreciated that the techniques introduced above anddiscussed in greater detail below may be implemented in any of numerousways, as the techniques are not limited to any particular manner ofimplementation. Examples of details of implementation are providedherein solely for illustrative purposes. Furthermore, the techniquesdisclosed herein may be used individually or in any suitablecombination, as aspects of the technology described herein are notlimited to the use of any particular technique or combination oftechniques.

IV. Further Descriptions of Techniques for Controlling Optical Behaviorof a Multi-View Display Using One or More Blurring Transformations

FIG. 1A shows an illustrative system 100 for generating actuationsignals for controlling a multi-view display and controlling themulti-view display using the generated actuation signals, in accordancewith some embodiments of the technology described herein. As shown inFIG. 1A, computing device(s) 104 is/are configured to generate actuationsignals and provide the generated actuation signals to electro-opticinterface circuitry 109, which uses the provided actuation signals(sometimes termed “actuation patterns”) to generate display interfacesignals and drive the multi-view display 111 using the generated displayinterface signals.

As shown in the illustrative embodiment of FIG. 1A, multi-view display111 comprises a front layer 111 a and a back layer 111 b. In someembodiments, layers 111 a and 111 b may both be active layers. In otherembodiments, front layer 111 a may be an active layer and back layer 111b may be a passive layer or vice versa. Non-limiting examples of anactive layer include a single layer LCD screen, a multi-layer LCDscreen, a layer comprising light emitting diodes (LEDs), a fluorescentor organic LED (OLED) backlight, an OLED layer, a layer comprising oneor more electronically focusable lenses, and multilayer polarizationrotators. An active layer may include one or multiple active opticalelements that may be electronically controlled. Non-limiting example ofsuch active optical elements include pixels, transistors, light emittingdiodes, color filters, liquid crystals, and/or any other electronicallyactuated components configured to emit and/or aid in emitting light orconfigured to selectively block and/or aid in selectively blockinglight. Non-limiting examples of a passive layer includes a polarizer, adiffuser, a brightness-enhancing film, a layer having a coating, a waveretarders, a color filter, a holographic layer, a parallax barrierlayer, and a lenslet array. It should be appreciated that the front andback layers 111 a and 11 b may include any other arrangement of opticalelements creating a linear or nonlinear parameterization of ray space.In embodiments where the layers 111 a and 111 b are active layers, thelayers 111 a and 111 b may comprise the same number of active opticalelements or a different number of active optical elements, as aspects ofthe technology described herein are not limited in this respect.

As shown in FIG. 1A, computing device(s) 104 generate(s) actuationsignals 108 a and 108 b used for controlling the optical behavior oflayers 111 a and 111 b of multi-view display 111. Computing device(s)104 provide(s) actuation signals 108 a to first electro-optic interfacecircuitry 109 a that, in response to receiving actuation signals 108 a,generates display interface signals 110 a to drive the front layer 111a. The display interface signals 110 a may comprise a display interfacesignal for each of one or more (e.g., all) of the optical elements infront layer 111 a. Actuation signals 108 a may comprise an actuationsignal for each of one or more (e.g., all) of the optical elements infront layer 111 a. Computing device(s) 104 also provide actuationsignals 108 b to second electro-optic interface circuitry 109 b that, inresponse to receiving actuation signals 108 b, generates displayinterface signals 110 b to drive the back layer 111 b. The displayinterface signals 110 b may comprise a display interface signal for eachof one or more (e.g., all) of the optical elements in back layer 111 b.Actuation signals 108 b may comprise an actuation signal for each of oneor more (e.g., all) of the optical elements in front layer 111 b. Amulti-view display is not limited to including only two layers, asillustrated in the illustrative embodiment of FIG. 1A and may includeany suitable number of layers including any suitable number of activelayers (e.g., 0, 1, 2, 3, 4, 5, etc.) and/or any suitable number ofpassive layers (e.g., 0, 1, 2, 3, 4, 5, etc.), as aspects of thetechnology described herein are not limited in this respect. Inembodiments where a multi-view display includes N active layers (where Nis an integer greater than two), the computing device(s) 104 may beconfigured to generate N sets of actuation signals and provide them toelectro-optical circuitry 109 that, in response generates N sets ofdisplay interface signals and uses the generated sets of displayinterface signals to drive the N active layers of the multi-viewdisplay.

In some embodiments, computing device(s) 104 may include one or multiplecomputing devices each being of any suitable type. Each computing devicemay include one or multiple processors. Each processor may be a centralprocessing unit (CPU), a graphics processing unit (GPU), a digitalsignal processor (DSP), an FPGA, an ASIC, any other type of hardwareprocessor, or any suitable combination thereof. When computing device(s)104 include multiple computing devices, the multiple computing devicesmay be located at one physical location or may be distributed amongdifferent physical locations. The multiple computing devices may beconfigured to communicate with one another directly or indirectly.

In some embodiments, including the illustrative embodiment shown in FIG.1A, computing device(s) 104 may be configured the generate actuationsignals (e.g., actuation signals 108 a and 108 b) based on: (a)information 105 specifying a desired light field to be reproduced bymulti-view display 111; (b) information 106 specifying of one or moreblurring transformations; and (c) information 107 specifying a model ofthe multi-view display 111. The computing device(s) 104 may generateactuation signals based on these inputs by using software 103 encodingone or more optimization algorithms for solving one or more optimizationproblems to obtain actuation signals based on these inputs. The software103 may comprise processor instructions that, when executed, solve theoptimization problem(s) to obtain actuation signals based on theabove-described inputs. The software 103 may be written in any suitableprogramming language(s) and may be in any suitable format, as aspects ofthe technology described herein are not limited in this respect.

Accordingly, in some embodiments, the actuation signals 108 a and 108 bmay be obtained as solutions to an optimization problem that isformulated, at least in part, by using: (a) information 105 specifying adesired light field to be reproduced by multi-view display 111; (b)information 106 specifying of one or more blurring transformations; and(c) information 107 specifying a model of the multi-view display 111.Examples of such optimization problems and techniques for generatingsolutions to them are described herein including with reference to FIGS.2-15.

Accordingly, in some embodiments, the content generated by multi-viewdisplay 111 may be obtained by solving at least one optimization problem(e.g., by one or more optimization algorithms including, for example,one or more iterative optimization algorithms). As such, multi-viewdisplay 111 may be referred to as an “optimized display.” An optimizeddisplay may be any display that generates content obtained by solving atleast one optimization problem.

In some embodiments, information 105 specifying a desired light field tobe reproduced by multi-view display 111 may include one or multiplescene views. The scene views may be of a natural scene or syntheticscene, and may be representative of a naturally occurring light field orof a light field that may not bear much resemblance to a naturallyoccurring light field. The latter case could correspond, by way ofexample and not limitation, to a scene having multiple distinct viewsshowing essentially independent two-dimensional content in each view. Insome embodiments, each scene view may correspond to a respectiveposition of a viewer of the multi-view display apparatus.

In some embodiments, the information 105 specifying one or more sceneviews may include an image (e.g., a PNG file, a JPEG file, or any othersuitable representation of an image) for each of one or more (e.g., all)of the scene views. The image may be a color image or a grayscale imageand may be of any suitable resolution. In some embodiments, the image ofa scene view may be generated by 3D generation software (e.g., AUTOCAD,3D STUDIO, SOLIDWORKS, etc.). The information 105 specifying the sceneviews may specify any suitable number of views (e.g., at least two, atleast ten, at least fifty, at least 100, at least 500, between 2 and1000, between 10 and 800, or in any other suitable combination of theseranges), as aspects of the technology provided herein are not limited inthis respect.

In some embodiments, information 106 specifying of one or more blurringtransformations may comprise any suitable data (e.g., numerical values)embodying the blurring transformation. The data may be stored in one ormore data structure(s) of any suitable type, which data structure(s) maybe part of the representation. Additionally or alternatively, theinformation specifying a blurring transformation may includeprocessor-executable instructions (e.g., software code in any suitableprogramming language, one or more function calls to one or moreapplication programming interfaces and/or software libraries, etc.)that, when executed, apply the blurring transformation to an image(e.g., by operating on a data structure encoding the image). It shouldbe appreciated that information 106 may specify one or multiple blurringtransformations in any suitable way, as aspects of the technologydescribed herein are not limited in this respect. The information 106may specify blurring transformations of any suitable type including anyof the types of blurring transformations described herein.

In some embodiments, information 107 specifying a model of themulti-view display 111 may include information characterizing one ormore physical characteristics of the multi-view display 111. Information107 may include information about any physical characteristics of themulti-view display 111 that influence the way in which the multi-viewdisplay generates images.

For example, in some embodiments, information 107 may includeinformation indicating a distance between the front layer and the backlayer, a relative location of the front layer to the back layer,resolution of the front layer, resolution of the back layer, size of thefront layer, size of the back layer, information about the response ofany color filters in the front layer and/or the back layer, arepresentation of spectral cross-talk between color channels of thefront layer and the back layer and/or any other suitable informationcharacterizing one or more physical characteristics of the multi-viewdisplay.

In some embodiments, multi-view display 111 may include one or moremultiplicative panel layers (e.g., one or more LCD panels withintegrated polarizers, as well as liquid crystal on silicon (LCOS) anddigital micro-mirror devices (DMD) or other electromechanical devices),and information 107 may include information indicating the effect of themultiplicative panel layer(s) on light passing through layers of themulti-view display 111. In some embodiments, multi-view display 111 mayinclude one or more additive panel layers (e.g., optically combinedLCDs, OLEDs, and LED elements), and information 107 may includeinformation indicating the effect of the additive panel layer(s) onlight passing through layers of the multi-view display 111. In someembodiments, multi-view display 111 may include one or morepolarization-rotating layers (e.g., LCD panels without polarizers), andinformation 107 may include information indicating the effect of thepolarization-rotating layers on light passing through layers of themulti-view display 111.

In some embodiments, information 107 may include information indicatingthe effect of one or multiple projection systems part of multi-viewdisplay 111. In some embodiments, information 107 may includeinformation indicating perspective effects of multi-view display 111,which effects may be representable as on-axis and off-axis projections.In some embodiments, information 107 may include a representation ofgenerally non-uniform sub-pixel tiling patterns, associated withreproducing various color channels in various layers. In someembodiments, information 107 may include a representation of spectralcross talk between red, green, blue, or other color channels. In someembodiments, information 107 may include a representation of theeffective minimum and maximum intensity levels attainable by the displayelements. In some embodiments, information 107 may include informationcharacterizing non-linear response characteristics (if any) of anymultiplicative and/or additive display elements in multi-view display111. In some embodiments, information 107 may include information aboutperturbations in position of one or more components of multi-viewdisplay 111 (e.g., as a consequence of manufacturing). In someembodiments, information 107 may include information about physicalmovements of display element positions (e.g., when the multi-viewdisplay 111 includes one or more motorized elements). In someembodiments, information 107 may include a representation of thetime-domain dynamics of optical elements in the multi-view display 111.By way of example and not limitation, said time-domain dynamics maycharacterize pixel state rise and fall time.

In some embodiments, information 107 may include a representation ofconstraints in the electro-optical interface circuitry 109 associatedwith transforming the actuation signals provided to display interfacesignals. By way of example and not limitation, the constraintsrepresented may reflect the allowable subsets of pixel states that maybe updated in a given clock cycle. By way of example and not limitation,it is possible to use a subset of row and column drivers, so that asubset of pixels can be updated at a rate that is higher than theequivalent full-refresh frame rate of the display element. Furthernon-limiting examples of display driver circuitry constraints that maybe represented include constraints reflecting the allowable precisionwith which values may be assigned to a particular pixel or set ofpixels. By way of example and not limitation, said pixel states may bespecified as some number of bits per color channel per pixel.

In some embodiments, information 107 may include informationcharacterizing one or more passive optical phenomena associated with themulti-view display 111. For example, in some embodiments, multi-viewdisplay 111 may include one or more passive layers (different fromlayers 111 a and 111 b), and information 107 may include informationcharacterizing the effects of the passive layer(s) on light passingthrough layers of the multi-view display 111. Such passive layers mayinclude one or more optical diffusers, one or more reflective elementsincluding specular and diffuse reflective elements, one or more opticalfilms, one or more lenslet arrays, one or more holographic layers (e.g.,diffractive holographic backlights). Such passive layers may be locatedin front of, in between two of, or behind any of the active layers inthe multi-view display 111. Additionally or alternatively, information107 may include information characterizing diffractive effects betweenoptical elements, for example, due to pixel aperture patterns,wavelength-dependent effects of any optical films, wavelength-dependenteffects of wave retarders (e.g., ½ wave plates), angle-dependentintensity responses including, for example, angle-dependent brightness,and contrast and/or gamma characterizations.

In some embodiments, information 107 may comprise a mapping betweenactuation signals used to drive a multi-view display and the displayviews generated by the multi-view display in response to the actuationsignals. The mapping may be generated using (and, as such, may representand/or reflect) any of the information described above as being part ofinformation 107. For example, the mapping may be generated using:information characterizing one or more physical characteristics of themulti-view display 111; information indicating a distance between thefront layer and the back layer, a relative location of the front layerto the back layer, resolution of the front layer, resolution of the backlayer, size of the front layer, size of the back layer, informationabout the response of any color filters in the front layer and/or theback layer, a representation of spectral cross-talk between colorchannels of the front layer and the back layer; information indicatingthe effect of the multiplicative, additive, and/or polarization rotatingpanel layer(s) on light passing through layers of the multi-view display111; information indicating the effect of one or multiple projectionsystems part of multi-view display 111; information indicatingperspective effects of multi-view display 111; representation ofgenerally non-uniform sub-pixel tiling patterns, associated withreproducing various color channels in various layers; a representationof spectral cross talk between red, green, blue, or other colorchannels; a representation of the effective minimum and maximumintensity levels attainable by the display elements; informationcharacterizing non-linear response characteristics of any multiplicativeand/or additive display elements in multi-view display 111; informationabout perturbations in position of one or more components of multi-viewdisplay 111; information about physical movements of display elementpositions (e.g., when the multi-view display 111 includes one or moremotorized elements; a representation of the time-domain dynamics ofoptical elements in the multi-view display 111; constraints in theelectro-optical interface circuitry 109 associated with transforming theactuation signals provided to display interface signals; informationcharacterizing one or more passive optical phenomena associated with themulti-view display 111; information characterizing diffractive effectsbetween optical elements; and/or any information about any physicalcharacteristics of the multi-view display 111 that influence the way inwhich the multi-view display generates images.

In some embodiments, the mapping between actuation signals used to drivea multi-view display and the display views generated by the display maybe generated (e.g., computed), and stored for subsequent use, andaccessed when they are to be used. In such embodiments, the mappings maybe stored in any suitable format and/or data structure(s), as aspects ofthe technology described herein are not limited in this respect. In someembodiments, the mapping may be generated and used right away, withoutbeing stored.

In some embodiments, the mapping may be generated using one or moresoftware packages. The software package(s) may take as input and/orparameters any of the above described information 107 to generatedisplay views from actuation signals. For example, in some embodiments,the mapping may be generated using a rendering package or framework(e.g., 3D Studio, Blender, Unity, ThreeJS, NVIDIA Optix, POVRay, orcustom or other packages, which may make use of various graphicsframeworks such as OpenGL, OpenGL ES, WebGL, Direct3D, CUDA, orgeneral-purpose CPU libraries) in rendering a model of the display inthe state corresponding to the use of the actuation signals, using acamera projection to obtain the view from the particular view locationof interest. The projection may be a perspective projection, an off-axisprojection, or an orthographic projection.

In embodiments where the actuation signals result in light beingselectively emitted from the rear layer and light being selectivelyattenuated in the front layer, the rear layer may be rendered as a planetextured with a first actuation signal, followed by a rendering of thefront layer as a plane textured with a second actuation signal, blendedwith the rendering of the rear layer using multiplicative blending. Insuch embodiments, performing a rendering of the scene using a cameraprojection whose viewpoint coincides with the desired location of thedisplay viewpoint may result in the computation of the associateddisplay view. In some embodiments where the display model is morecomplex (e.g., involving a model of reflective layers, diffuse layers,spectral cross-talk between color channels, diffractive effects, orinternal reflections between layers) the mapping from the actuationsignals to the display views may be generated using optics modelingroutine or software (e.g., NVIDIA Optix, Maxwell, or custom-writtensoftware).

FIG. 1B shows an illustrative system 110 for generating patterns to beprinted on layers of a light field print and printing the generatedpatterns on the layers of the light field print, in accordance with someembodiments of the technology described herein. As shown in FIG. 1B,computing device(s) 113 is/are configured to generate actuation signalsand provide the generated actuation signals to a printing system 118,which prints the provided actuation signals (sometimes termed “actuationpatterns” or “target patterns”) on layers printed media, which arearranged into a layered passive display arrangement such as light fieldprint 120.

As shown in the illustrative embodiment of FIG. 1B, light field print120 comprises a front layer 120 a and a back layer 120 b. Each of theselayers may include one or more transparent film and/or other transparentmaterials on which generated actuation patterns may be printed byprinting system 118. Additionally, in some embodiments, light fieldprint 120 may include one or more other layers including, but notlimited to, one or more optical spacers, one or more diffusers, one ormore lenslet arrays, one or more holographic layers, one or more colorfilters, and/or one or more active backlights.

As shown in FIG. 1B, computing device(s) 113 generate(s) target patterns117 a and 117 b for depositing onto layers 120 a and 120 b. Computingdevice(s) 113 provide(s) the generated target patterns to printingsystem 118, which prints the target patterns onto the layers 120 a and120 b. The printing system 118 may be a laser toner-based printingsystem, laser drum-based printing system, an inkjet printing system, achromogenic or other photographic printing system, digital offsetprinting system, and/or any other type of printing system that may beused to print target patterns on one or more layers used to assemble alight field print.

A light field print is not limited to having only two layers, asillustrated in the illustrative embodiment of FIG. 1B, and may includeany suitable number of layers (e.g., 2, 3, 4, 5, 6, 7, etc.), as aspectsof the technology described herein are not limited in this respect. Inembodiments where a light field print includes N layers (where N is aninteger greater than two), the computing device(s) 113 may be configuredto generate N target patterns and provide them to printing system 118,which prints the generated target patterns on the N layers, which layersmay be subsequently assembled into a light field print.

In some embodiments, computing device(s) 113 may include one or multiplecomputing devices each being of any suitable type. Each computing devicemay include one or multiple processors. Each processor may be a centralprocessing unit (CPU), a graphics processing unit (GPU), a digitalsignal processor (DSP), an FPGA, an ASIC, any other type of hardwareprocessor, or any suitable combination thereof. When computing device(s)113 include multiple computing devices, the multiple computing devicesmay be located at one physical location or may be distributed amongdifferent physical locations. The multiple computing devices may beconfigured to communicate with one another directly or indirectly.

In some embodiments, including the illustrative embodiment shown in FIG.1B, computing device(s) 113 may be configured the generate targetpatterns (e.g., target patterns 117 a and 117 b) based on: (a)information 115 specifying a desired light field to be reproduced bylight field print 120; (b) information 114 specifying of one or moreblurring transformations; and (c) information 116 specifying a model ofthe printing process performed by printing system 118. The computingdevice(s) 113 may generate target patterns based on these inputs byusing software 112 encoding one or more optimization algorithms forsolving one or more optimization problems to obtain target patternsbased on these inputs. The software 112 may comprise processorinstructions that, when executed, solve the optimization problem(s) toobtain target patterns based on the above-described inputs. The software112 may be written in any suitable programming language(s) and may be inany suitable format, as aspects of the technology described herein arenot limited in this respect.

Accordingly, in some embodiments, the target patterns 117 a and 117 bmay be obtained as solutions to an optimization problem that isformulated, at least in part, by using: (a) information 115 specifying adesired light field to be reproduced by light field print 120; (b)information 114 specifying of one or more blurring transformations; and(c) information 116 specifying a model of the printing process performedby printing system 118. Examples of such optimization problems andtechniques for generating solutions to them are described hereinincluding below with reference to FIGS. 2-15.

In some embodiments, information 115 specifying a desired light field tobe reproduced by a light field print may include one or multiple sceneviews and, for example, may include any of the information describedabove in connection with information 105 in FIG. 1A. In someembodiments, information 114 specifying of one or more blurringtransformations may include any of the information described above inconnection with information 106 in FIG. 1A.

In some embodiments, information 116 specifying a model of the printingprocess performed by printing system 118 may include informationcharacterizing the printing process including, but not limited to, layergeometry information, color model information, print resolutioninformation, information specifying the type of printing system used,information characterizing how much ink bleed results from the printingprocess, information characterizing how much dot gain results from theprinting process, information indicating the maximum allowable inkdensity of the printing medium, information indicating the dot pitch ofthe prints generated by the printing process.

As may be appreciated from the foregoing discussion of FIGS. 1A and 1B,the techniques described herein may be applied to generating actuationpatterns for controlling active displays and to generating targetpatterns for passive displays (including printed and CNC manufacturedmaterials). Non limiting examples of applications of the techniquesdescribed herein are listed below. Applications of generating actuationsignals for controlling active elements include, but are not limited to,providing electronic displays that may be used at positions far from theviewer's eyes (e.g., producing an illusion of depth without requiringspecial eyewear or other hardware, producing multiple independent viewsof a scene, correcting for vision deficiencies of a viewer); providingelectronic displays that may be used at positions close from theviewer's eyes with or without an optical system combining display outputwith the surrounding existing light field (e.g., reproducingaccommodation cues in virtual reality or augmented realityapplications); providing electronic displays that may be used atpositions far from the viewer's eyes superimposed on the surroundingexisting light field using an optical system (e.g., a heads up displaysystem). Any of these types of displays may be used as part of mobiledevices (e.g., mobile phones, wearable devices and tablets),entertainment systems (e.g., home television displays, in-flightentertainment systems, and automotive entertainment systems), datavisualization systems, general-purpose computation systems and computeruser interfaces, digital signage including advertisement and informationkiosks, architectural preview displays, CAD workstation displays,automotive and avionics instrumentation displays, and any other suitabletypes of displays. Applications of generating actuation patterns forpassive display elements include, but are not limited to, producingfurniture, cabinetry, architecture and architectural decoration, stageand television sets, signage, advertisements and promotional materials,including signage with passive or active backlights, generating printedmaterial, for example, of less than 0.02 inches in thickness, CNCmachined, and multi-layer items. As described herein, in someembodiments, an optimization-based approach may be used to generateactuation signals for controlling one or more active layers and/or forgenerating target patterns for printing onto one or more layers oftransparent materials. In some embodiments, the optimization-basedapproach may be iterative.

In some embodiments, the approach may be as follows. First, at aninitialization stage, a first set of actuation signals (or targetpatterns in the printing context) is generated. This set of actuationsignals may be generated fresh or based on using one or morepreviously-obtained actuation signals. The first set of actuationsignals is then used to determine a first set of display views thatwould be generated by a multi-view display if it were driven by thefirst set of actuation signals, and the display views are compared tothe scene views (which specify the desired light field to be produced bythe multi-view display) to generate error views. A display view may begenerated for each scene view. The display views may be determined usinginformation about the physical characteristics of the multi-view display(e.g., information 107 described with reference to FIG. 1A). In theprinting context, the views may be determined using information aboutthe printing process (e.g., information 116 described with reference toFIG. 1B).

In some embodiments, the error views may be generated further based uponusing one or more blurring transformations (e.g., the same blurringtransformation for pairs of display and scene views). In someembodiments, prior to being compared to generate the error views, eachof the display views and scene views may be transformed by a suitableblurring transformation (e.g., as shown in FIG. 2). In some embodiments,when the blurring transformations applied to the display and scene viewsare identical and linear, the blurring transformation may be applied tothe error views instead of being applied to the display views and sceneviews.

In turn, the error views may be used to determine how to update thevalues of the first set of actuation signals to obtain a second set ofactuation signals (in order to reduce the error between the displayviews and the scene views). The second set of actuation signals are thenused to determine a second set of display views that would be generatedby a multi-view display if it were driven by the second set of actuationsignals, and a second set of error views is generated by comparing thesecond set of display views with the scene views. The second set oferror views is then used to determine how to update the values of thesecond set of actuation signals to obtain a third set of actuationsignals in order further reduce the error between the display vies andthe scene views. This iterative process may be repeated until the errorbetween the display views and the scene views falls below apredetermined threshold, a threshold number of iterations has beenperformed, a threshold amount of time has elapsed, or any other suitablestopping criteria has been satisfied.

Although the above illustrative iterative optimization technique wasdescribed with respect to generating actuation signals for controllingactive displays, it should be appreciated that analogous techniques maybe used to generate target patterns for printing onto transparent layersused to assemble light field prints. Similarly, in descriptions below,the optimization techniques described with reference to FIGS. 2-14 maybe applied to generating not only actuation signals for active displays,but also for generating target patterns for manufacturing light fieldprints.

FIG. 2 is an illustrative block diagram 200 of the processing performedto generate actuation signals for controlling a multi-view display, inaccordance with some embodiments of the technology described herein. Inparticular, FIG. 2 illustrates a step of an iterative optimizationtechnique for identifying the set of actuation signals based on acomparison between a set of display views 202 of a multi-view display201 having layers 201 a and 201 b, which display views are denoted byd_(k) (k=1, . . . , N) with N representing the number of views, and aset of corresponding scene views 204, which are denoted by s_(k) (k=1, .. . , N), of a virtual scene 203. The scene views may of any suitabletype including the types described herein. For example, the scene viewsmay be of a natural scene or synthetic scene, and may be representativeof a naturally occurring light field or of a light field that may notbear much resemblance to a naturally occurring light field. The lattercase could correspond, by way of example and not limitation, to a scenehaving multiple distinct views showing essentially independenttwo-dimensional content in each view.

In some embodiments, there may be a one-to-one correspondence betweenthe display and scene views. In other embodiments, there may not be sucha one-to-one correspondence. For example, the scene views may correspondto the display views when moving in the horizontal direction only,whereas moving in the vertical direction, an ensemble of display viewsmay correspond to a single scene view. As another example, whencomparing scene and display views by moving in the horizontal direction,the scene view location may advance at some fraction of (e.g., half) therate as the rate of the display view location.

As shown in FIG. 2, blurring transformation(s) 208 may be applied to thedisplay views 202 and the scene views 204 and the resulting blurreddisplay views and blurred scene views may be compared to generate errorviews 212, denoted by e_(k) (k=1, N). In some embodiments, the sameblurring transformation may be applied to all display views and allscene views. In other embodiments, one blurring transformation T_(i) maybe applied to a display view d_(i) and a corresponding scene view s_(i)and a different blurring transformation T_(j) may be applied to anotherdisplay view d_(j) and its corresponding scene view s_(j). The blurringtransformation(s) 208 may include any of the types of blurringtransformations described herein.

The display views may be generated using a set of actuation signals 205,denoted by x_(k) (k=1, . . . , M), where M indicates the number ofactuation signals. The actuation signals 205 may be used in assigningstate to various layers of the multi-layer display 201 and/or asimulation of a multi-layer display 201 to generate the display viewsd_(k) 202 based on information specifying a model of the multi-layerdisplay 201, which information may include any of the information 107described with reference to FIG. 1A and, in some embodiments, mayinclude one or more mappings (previously generated and stored orspecified in software for “on-line” use) between actuation signals anddisplay views. Accordingly, in some embodiments, one or more mappingsmay be used to assign state to various layers of the multi-layer display201 based on actuation signals 205. In this sense, the mappings fromactuation signals 205 (i.e., x_(k) (k=1, . . . , M)) to display views202 (i.e., d_(k) (k=1, . . . , N)) may be formulated usingcharacterizations of the overall multi-layer display 201 including itsphysical light transport and passive or active display elements.Accordingly, the mappings from the actuation signals 205 to the displayviews 202 may depend on any information characterizing one or morephysical characteristics of the multi-view display 111 including, forexample, any of the information 107 described with reference to FIG. 1A.

In some embodiments, as may be appreciated from FIG. 2, at eachiteration of an optimization algorithm, the goal may be to update theactuation signals 205 based on the error views 212 to reduce the overallamount of error between blurred versions of the display views andblurred versions the scene views. This means that, the non-blurreddisplay view can (and in practice will) have a large amount ofhigh-frequency content, which is removed via the application of theblurring transformations 208. Put another way, an error function thatweights low-frequency content more significantly, may encourage theactuation signals to cause the multi-view display to generatehigh-frequency content since the high frequency content will not counttoward the error function as significantly. In some embodiments, theblurring transformation(s) 208 may encourage this by weightinglow-frequency content so that the error penalty is higher at in lowerfrequencies, and so that the error penalty is smaller at higherfrequencies.

Additional aspects of the optimization techniques which may be used togenerate actuation signals for controlling an active display (e.g.,multi-view display 111) or target patterns for manufacturing a lightfield print (e.g., light field print 120) are described below withreference to FIGS. 3-13.

FIG. 3 shows an example optimization problem 300 that may solved as partof generating actuation signals for controlling a multi-view displayand/or as part of generating patterns for printing on one or more layersof a light field print, in accordance with some embodiments of thetechnology described herein.

As shown in FIG. 3, the optimization problem 300 may be used todetermine the actuation signals x_(k) by minimizing (exactly orapproximately) the cost function g(e₁, . . . , e_(N)), subject to thelisted constraints, which include upper bounds u_(k) and lower boundsl_(k) on the actuation signals. Such constraints would be enforcedelement-wise. In the optimization problem 300, the functions ƒ_(k)( . .. ), k=1, . . . , N, represent the mapping from the actuation signalsx_(k) to the view error signals e_(k), such as the view error signalsshown in FIG. 2. In this sense, the functions ƒ_(k)( . . . ) generallyincorporate, for example, (1) the implicit mappings from the actuationsignals x_(k) to the display views d_(k); (2) the values of the desiredscene views s_(k); and (3) the blurring transformations and differencingfunctions shown in FIG. 2.

In some embodiments, the optimization problem 300 may be solved using aniterative gradient-based technique to obtain the actuation signalsx_(k), as is depicted schematically in FIG. 4. As illustrated, thegradient technique comprises using a gradient of the functions ƒ_(k)( .. . ) to iteratively update values of the actuation signals using anupdate rule.

FIG. 5 illustrates an example of an update rule 504 that may be used aspart of the gradient-based technique of FIG. 4 in some embodiments. Theupper and lower bounds u_(k) and l_(k) shown in FIG. 3, which constrainthe actuation signals x_(k), may be enforced by the update rule 504 bybeginning with a set of variables x_(k) 501 that are known to meet theconstraints, and dynamically selecting values α 502 and β 503 thatresult in a state evolution always satisfying these constraints.

FIG. 6 shows another optimization problem 600 that may solved as part ofgenerating actuation signals for controlling a multi-view display and/oras part of generating patterns for printing on one or more layers of alight field print, in accordance with some embodiments of the technologydescribed herein. The optimization problem 600 may be obtained byreplacing the upper and lower bounds in the optimization problem 300 bypenalty terms in the cost function. The penalty terms would be selectedso that the constraints are met as the state evolves or as the systemreaches steady-state. In the illustrative optimization problem 600, thepenalty terms are the penalty functions p_(k)(x_(k)).

In some embodiments, the optimization problem 600 may be solved by agradient-based iterative technique illustrated in FIG. 7. As shown inFIG. 7, this technique makes use of an update rule 701 and incorporatesa gradient of the penalty term. The update rule 701 may be any suitableupdate rule and, for example, may be the update rule 504 shown in FIG.5.

Still referring to FIG. 7, the following defines our notation for thefunctions ƒ_(j,k)( . . . ):ƒ_(j,k)(x _(j))=ƒ_(k)( . . . ,x _(j), . . . ).Accordingly, each ƒ_(j,k)( . . . ) is defined as being that functionobtained by beginning with ƒ_(k)( . . . ) and holding all but theargument in position j fixed. The particular fixed values of thosevariables not in position j would retain the previously-defined valuesof those variables, as defined elsewhere within the global problemscope. This definition is used without loss of generality andfacilitates discussion in the following section.

In some embodiments, the processing required for determining values ofactuation signals by solving one or more optimization problems (e.g., byfinding an exact or an approximate solution) may be performed in adistributed manner by multiple computing devices. Discussed below aretechniques, developed by the inventors, in which the optimizationalgorithms developed by the inventors may be distributed, in someembodiments. The topology of the distributed hardware and software isimplied by these descriptions.

In some embodiments, an optimization problem (e.g., such as optimizationproblems 300 and 600) may be “partitioned” (that is, a technique forsolving the optimization problem may be designed in a way thatfacilitates its implementation in a distributed environment) by holdinga subset of the actuation signals x_(k) constant and performing somenumber of iterations to optimizing the values of remaining subset ofactuation signals. After this point, a different subset of the actuationsignals x_(k) may be selected, and the process would be repeated untildesired values for the actuation signals are obtained.

FIG. 8 illustrates an optimization problem 800 formulated so as tofacilitate the distribution implementation of a gradient-based iterativeoptimization technique for solving the optimization problem 300. In someembodiments, a solution (exact or approximate—finding the global or alocal minimum) of the optimization problem 300 may be obtained bysequentially updating each of the actuation signals as shown in Table 1.

TABLE 1 Iterative technique for identifying a solution to optimizationproblem 300. 1. Choose j = 1. 2. Select initial values x_(j) consistentwith the upper and lower constraints listed in FIG. 3. 3. Find a globalor local minimum of the optimization problem listed in FIG. 8. using anyof the techniques described herein, or compute a finite number ofiteration steps toward an acceptable solution. The obtained value ofx_(j) would be used implicitly by all other functions f_(j,k)(x_(j))until the value of x_(j) is otherwise re-defined. 4. Choose the nextinteger value of j between 1 and M, returning eventually from M to 1. 5.Go to step 3.

An iterative gradient-based optimization algorithm that could be used,in some embodiments, to finding a local or global minimum of theoptimization problem 800 shown in FIG. 8, or alternatively that could beused in taking a finite number of iteration steps toward such asolution, is depicted schematically in FIG. 9.

In some embodiments, to further partition and distribute computation,additional mathematical structure in the formulation of the optimizationproblem 800 may be utilized. For example, with reference to FIG. 8,selecting the functions ƒ_(j,k) (x_(j)) asƒ_(j,k)(x _(j))=L _(k)(h _(j,k)(x _(j))−s _(k)),with each function h_(j,k)( . . . ) and L_(k) ( . . . ) being a linearmap, would result in the further decomposition of the optimizationalgorithm, as shown schematically in FIG. 10.

In FIG. 10, a superscript asterisk denotes the adjoint map, which in thecase of matrices would reduce to the matrix transpose. Note that in thisformulation, the variables s_(k) 1001 may represent the scene views, thefunctions h_(j,k) ( . . . ) 1002 may represent the mappings from theactuation signals x_(j) 1003 to the display views d_(k) 1004, and thefunctions L_(k)( . . . ) 1005 may represent the linear maps implementinga blurring transformation (which, in some embodiments, may be realizedexplicitly as convolution with a blur kernel). Linearity of h_(j,k) ( .. . ) and L_(k)( . . . ), for example, arises naturally in optimizingactuation signals in a multi-layer display consisting of multiplicativelayers, wherein the overall mapping from the set of all actuationsignals 1003 to the individual display views d_(k) 1004 is amulti-linear map.

As shown in FIG. 10, the functions g_(j,k)(e_(k)) (1006) are taken toindividually sum to an overall cost term as listed in the optimizationproblem 800 of FIG. 8, where:g(e ₁ , . . . ,e _(N))=g _(j,1)(e ₁)+ . . . +g _(j,N)(e _(N)).In this sense, the individual functions g_(j,k)(e_(k)) may be linear ornonlinear penalty functions, whose gradients would be computed asdepicted in FIG. 10. It is straightforward to show, for example, thatchoosing g_(j,k)(e_(k))=∥e_(k)|_(γ) ^(γ) would result in an algorithmwhere a local minimum of ∥g_(j,k)(e_(k))∥_(γ) ^(γ) is obtained, withindicating the γ-norm.

In some embodiments, quadratic penalty functions g_(j,k)(e_(k)) may beemployed. When the functions g_(j,k)(e_(k))=∥e_(k)∥_(γ) ^(γ) arequadratic (e.g., γ=2) and if the intent is to enforce a lower bound onthe actuation signals x_(j) corresponding to non-negativity, amultiplicative update rule may be used in some embodiments. On thesurface, this appears similar to multiplicative update rules that aresometimes used in solving nonnegative matrix factorization problems.However, the techniques described in such embodiments utilize the moregeneral property of linearity in the functions composing ƒ_(j,k)(x_(j))=L_(k)(h_(j,k)(x_(j))−S_(k)), which corresponds tomulti-linearity in the mappings from the ensemble of actuation signalsx_(j) for the ensemble of display layers to error views e_(k). This is afar less restrictive assumption than is used with weighted orun-weighted nonnegative matrix factorization, and it is specificallyuseful for formulating algorithms for taking advantage of blurringtransformations.

FIGS. 11 and 12 illustrate two techniques, which may be used in someembodiments, for finding a local or global minimum (or taking one ormore steps toward such a solution) of the optimization problem 800 shownin FIG. 8, utilizing a multiplicative update rule enforcingnon-negativity of the actuation signals x_(j). As shown in FIG. 11, thenumerator term 1101 and the denominator 1102 terms resulting fromvarious sub-computations are combined additively and the individual sumsare divided. As shown in FIG. 12, in each sub-computation 1201, adivision 1203 among various signals is performed first, and a convexcombination 1204 of the results of the individual sub-computations 1205is taken, by way of example and not limitation, corresponding to aweighted average with the weights being nonnegative and summing to 1. Ageneral form of the multiplicative update rules 1103 and 1202 utilizedin FIGS. 11 and 12, respectively, is depicted schematically in FIG. 13.

Additional aspects of the FIGS. 2-13 may be appreciated through thefollowing further explanation of certain diagram notations used therein.Arrows may represent the direction of signal flow with time. Signal flowmay correspond to the synchronous or asynchronous passing of variables,which may generally take scalar values, or vector values denoting forexample the flow of image data or collections of image data. Thecircled + symbol indicates generally vector addition or subtraction ofinput signals (e.g., as shown in FIG. 2). For any inputs to a circled +symbol having negative signs written at the input, these input signalsare negated. After possible negation, all signals are summed to form theoutput signal. A dot on a signal line indicates signal duplication(e.g., as indicated after the output of “State storage” in FIG. 5). Thesymbol ∇ denotes the gradient of a function (e.g., in FIG. 10 the squareblock that contains this symbol refers to applying the negative of thegradient of the functional g_(j,k)( . . . ) to the signal e_(k), whichis the input to that block). In FIGS. 11 and 12, the circled ÷ symbolindicates element-wise division of generally vector-valued signals andthe boxed symbols indicate application of the labeled linear map to theassociated input signal. In FIG. 13, the circled X symbol indicateselement-wise multiplication of generally vector-valued signals.

Table 2 illustrates pseudo-code that describes aspects of an iterativegradient-based optimization technique may be used to obtain a local or aglobal solution to an optimization problem in order to generate valuesfor actuation signals, in accordance with some embodiments.

TABLE 2 Pseudo-code describing aspects of an iterative gradient-basedoptimization technique for generating actuation signals, in accordancewith some embodiments. 0. (Initialize)  We denote the vector ofactuation signals for a first layer as x₁ and the vector of actuationsignals for a second layer as x₂. Each set of actuation signals has acorresponding lower bound vector l_(i) and upper bound vector u_(i).Perform the following initialization:   a. Initialize the elements of x₁to a value greater than 0 for which l₁ ≤ x₁ ≤ u₁.   b. Initialize theelements of x₂ to a value greater than 0 for which l₂ ≤ x₂ ≤ u₂. 1.(Compute gradient step for a first layer)  For each view image, indexedk = 1, . . . , N:   a. Compute view k of current display state, denotedd_(k). The view of the display state d_(k) will generally depend on theactuation signals x₁ and x₂.   b. Compute corresponding view k of scene,denoted s_(k).   c. Compute error view as e_(k) = BL_(s)(s_(k)) −BL_(d)(d_(k)). The functions BL_(s) and BL_(d) are band-limitingtransformations as discussed above.   d. Compute the gradient stepcontribution q_(k) ⁽¹⁾ due to view kas: q_(k) ⁽¹⁾ = α [PROJ^((k)) _(v)_(k) _(→x) ₁ (BL^(*) _(d)(e_(k)))] * [PROJ^((k)) _(x) ₂ _(→x) ₁ (x₂)].Referring to this equation:  (1) PROJ^((k)) _(x) ₂ _(→x) ₁ (x₂) denotesthe perspective projection of x₂ from the coordinate system of a secondlayer to the coordinate system of a first layer, with the camera centerfor the projection being the location of viewpoint k.  (2) BL^(*) _(d)denotes the adjoint operator corresponding to the band-limitingtransform BL_(d).  (3) PROJ^((k)) _(v) _(k) _(→x) ₁ (BL^(*) _(d)(e_(k)))denotes the perspective projection of BL^(*) _(d)(e_(k)) from thecoordinate system of error view k to the coordinate system of a firstlayer, with the camera center for the projection being located atviewpoint k.  (4) The symbol * denotes element-wise multiplication.  (5)The variable α denotes the step size. 2. (Update actuation signals for afirst layer)   a. Perform the following assignment:     $x_{1}:={x_{1} + {\sum\limits_{k = 1}^{N}q_{k}^{(1)}}}$   b. Enforceequality constraints, hard-limiting x₁ to fall in the range l₁ ≤ x₁ ≤u₁. 3. (First layer loop) Go to step 1, and loop some finite number oftimes. 4. (Compute gradients for a second layer)  For each view image,indexed k = 1, . . . , N:   e. Compute view k of current display state,denoted d_(k). The view of the display state d_(k) will generally dependon the actuation signals x₁ and x₂.   f. Compute corresponding view k ofscene, denoted s_(k).   g. Compute error view as e_(k) = BL_(s)(s_(k)) −BL_(d)(d_(k)). The functions BL_(s) and BL_(d) are band-limitingtransformations as discussed above.   h. Compute the gradient stepcontribution q_(k) ⁽²⁾ due to view kas: q_(k) ⁽²⁾ = α [PROJ^((k)) _(v)_(k) _(→x) ₂ (BL^(*) _(d)(e_(k)))] * [PROJ^((k)) _(x) ₁ _(→x) ₂ (x₂)].Referring to this equation:  (1) PROJ^((k)) _(x) ₁ _(→x) ₂ (x₁) denotesthe perspective projection of x₁ from the coordinate system of a firstlayer to the coordinate system of a second layer, with the camera centerfor the projection being the location of viewpoint k.  (2) BL^(*) _(d)denotes the adjoint operator corresponding to the band-limitingtransform BL_(d).  (3) PROJ^((k)) _(v) _(k) _(→x) ₂ (BL^(*) _(d)(e_(k)))denotes the perspective projection of BL^(*) _(d)(e_(k)) from thecoordinate system of error view k to the coordinate system of a secondlayer, with the camera center for the projection being located atviewpoint k.  (4) The symbol * denotes element-wise multiplication.  (5)The variable α denotes the step size. 5. (Update actuation signals for asecond layer)   a. Perform the following assignment:     $x_{2}:={x_{2} + {\sum\limits_{k = 1}^{N}q_{k}^{(2)}}}$   b. Enforceequality constraints, hard-limiting x₂ to fall in the range l₂ ≤ x₂ ≤u₂. 6. (Second layer loop) Go to step 4, and loop some finite number oftimes. 7. (Overall loop) Go to step 1, and loop the overall iterationsome finite number of times until completion.

FIG. 14 illustrates simulated views generated by a multi-view display inaccordance with some embodiments of the technology described herein.Images 1401 show two views of a multi-view light field image comprising15 views, which 15 views are specified as the input to all comparedmethods. Images 1402 show the results from running methods previouslyknown to those skilled in the art which utilize nonnegative matrixfactorization (NMF), or methods that reduce to NMF in the case of twolayers. Images 1403 and 1404 show the performance achieved by some usingtechniques described herein, which utilize a perceptually-inspired costfunction taking advantage of finite view bandwidth. Shown in 1403 and1404 are simulations of two extreme views along the horizontal parallaxdirection of a 3×5 (15 view) light field with 10 degree horizontal FOV,presented on a simulated 47 cm×30 cm, two-layer display with a layerseparation of 1.44 cm, at a viewer distance of 237 cm. Images 1403 and1404 compare the performance of one embodiment of the disclosed methodsas the scene brightness is varied. Light field data and displayconfiguration were obtained from [G. Wetzstein. Synthetic Light FieldArchive. http://web.media.mit.edui˜gordonw/SyntheticLightFields/.Accessed Aug. 12, 2015.]. For each approach 1402-1404, the requiredincrease in display backlight brightness is listed, indicating that alarge increase in backlight efficiency can be achieved as compared to atraditional barrier-based parallax display. Note that all presentedresults show performance for single-frame, non-time-multiplexeddisplays, in contrast to time multiplexed work that has been previouslydemonstrated. Depicted results are filtered to simulate observation bythe human visual system, excluding magnified detail view 1405.

FIG. 15 is a flowchart of an illustrative process 1500 for generatingactuation signals to control optical behavior of a multi-view displayapparatus in accordance with some embodiments of the technologydescribed herein. Process 1500 may be performed by any suitabledevice(s). For example, process 1500 may be performed by one orcomputing device(s) coupled to and/or part of the multi-view display.For example, process 1500 may be performed by computing device(s) 104described with reference to FIG. 1A.

Process 1500 begins at act 1502, where a plurality of scene views may beobtained. Each of the plurality of scene views may correspond to alocation of a viewer of the multi-view display. The scene views mayspecify a desired light field to be generated by the multi-view display.As described herein, the scene views may be of a natural or a syntheticscene. Each scene view may comprise a grayscale and/or a color image ofany suitable resolution for each of one or more (e.g., all) of the sceneviews. Any suitable number of scene views may be obtained at act 1502(e.g., at least two, at least ten, at least fifty, at least 100, atleast 500, between 2 and 1000, between 10 and 800, or in any othersuitable combination of these ranges), as aspects of the technologyprovided herein are not limited in this respect.

In some embodiments, the scene views may be obtained by accessing and/orreceiving one or more images from at least one image source (e.g.,accessing stored images, receiving images from another applicationprogram or remote computing device). In some embodiments, the sceneviews may be obtained by first obtaining a description of a 3D scene(e.g., a 3D model of a scene) and then generating, as part of process1500, the scene views based on the obtained description of the 3D scene.

Next, process 1500 proceeds to act 1504, where information specifying amodel of the multi-view display may be obtained. This information mayinclude any information about any physical characteristics of themulti-view display apparatus, which may influence the way in which themulti-view display generates images. The information obtained at act1504 may include, for example, any of information 107 described withreference to FIG. 1A.

In some embodiments, the information obtained at act 1504 may includedata specifying physical characteristics of the multi-view displaynumerically (e.g., using one or more values stored in one or more datastructures of any suitable type) such that these data may be used togenerate display views based on a set of actuation signals as part of aniterative optimization technique for identifying actuation signals(e.g., as described with reference to FIGS. 2-13). In some embodiments,the information obtained at act 1504 may be encoded in software code.The software code may also be used to generate display views based on aset of actuation signals as part of an iterative optimization techniquefor identifying actuation signals. In some embodiments, when suchsoftware code is executed it may be used to transform parameters (e.g.,actuation signals, display views or other images, other variables) basedon the physical characteristics embodied in the software code.

Next, process 1500 proceeds to act 1506, where information specifying atleast one blurring transformation may be obtained. The informationspecifying the at least one blurring transformation may specify one ormultiple blurring transformations and may include information of anysuitable type including, for example, any of information 106 describedwith reference to FIG. 1A.

Next, process 1500 proceeds to act 1508, where a plurality of actuationsignals may be generated based on the plurality of scene views obtainedat act 1502, information specifying a model of the multi view displayapparatus obtained at act 1504, and information specifying at least oneblurring transformation obtained at act 1506. This may be done in any ofthe ways described herein and, for example, by using an iterativeoptimization techniques described with reference to FIGS. 2-13.

Next, process 1500 proceeds to act 1510, where the actuation signalsgenerated at act 1508 may be used to control the multi-view display.This may be done in any suitable way. For example, in some embodiments,the generated actuation signals may be provided to electro-opticalinterface circuitry (e.g., circuitry 109 described with reference toFIG. 1A), and the electro-optical interface circuitry may drive themulti-view display based on the provided actuation signals. After act1510, process 1500 completes.

It should be appreciated that the techniques described herein that useblurring transformations may be used in applications where the blurringtransformation(s) do not relate to a perceptual effect (e.g., that ofthe human visual system) but rather relate to some band-limited effectin the medium receiving the light or other electromagnetic wave outputfrom the display. In such applications, the light or electromagneticwave emitted from the display might not typically be designed forconsumption by a human eye, but rather by another physical medium orbiological tissue. Non-limiting examples of such applications include:

-   -   The use of band-limitedness in optimized displays for        photolithography and stereolithography in 3D printing (e.g.,        where an optimized display may be used for emitting into a        photosensitive resin). Here the band-limitedness would embody        the lower limit on resolvable dot size in the resin.    -   The use of band-limitedness in optimized displays for        photographically exposing two-dimensional materials (e.g., used        in a photogenic printing process or other photographic printing        device). Here the band-limitedness would embody the lower limit        on the resolvable dot size on the photographic medium.    -   The use of band-limitedness in optimized displays for radiating        biological tissue using light or other energy electromagnetic        waves. The band-limitedness may embody the minimum volume of        tissue that can independently be affected by radiation due to        thermal or other effects.        V. Further Descriptions of Multi-View Display Arrangements

The techniques for generating actuation signals for controllingmulti-view displays may be used with numerous types of multi-view 3Ddisplays described herein. The following includes a description of sometypes of multi-view 3D displays developed by the inventors, and relatedtechniques. Some of these 3D displays may have favorable characteristicsfor synthesizing light fields in accordance with some of theoptimization techniques described herein including, for example, any ofthe techniques described with reference to FIGS. 1-15.

The goal of an optimized light field display is essentially to exploitredundancy caused by both the structure of the data to be displayed andexternal factors including the response of the human visual system anddisplay optics, in order to represent light field images optically for ahuman observer. When considered through simple linear analysis suchsystems seem, at first glance, to violate simple counting arguments—inthe case of light field synthesis the display appears to create moreindependent rays than there are independent image elements.

In fact, such displays produce an output with the same number of degreesof freedom as the display hardware. The bandwidth or algebraic rank ofthe output will be limited by the degrees of freedom of the displayhardware. Another way to see this is by observing that the number offree parameters in the display system scales with the number of imageelements, but the parameter space of the display system can be largewhen a suitable non-linear mapping between pixel states (or equivalentlythe actuation signals driving the pixel states) and output light rayintensities is created. This insight gives us a new means foridentifying display systems that will be amenable to driving withoptimization. Systems with this property may be referred to asparametric display systems.

This insight is further illustrated in FIGS. 16A and 16B, whichrepresent means of addressing a linear space. In the depicted schematic,boxes may represent pixels on a screen, light rays in a light field, orother abstract addressable quantities. FIG. 16A shows a directaddressing scheme wherein boxes are addressed by a Cartesian coordinate.In this addressing scheme the number of parameters required to addressthe space is equal to the number of dimensions of the space. FIG. 16Bshows an indirect, or parameterized addressing scheme wherein anonlinear function C(t) is used to address boxes. In this addressingscheme the number of parameters required to address the space is fewerthan the number of dimensions of the space. This advantageous situationmotivates some embodiments of the technology described herein.

An important benefit of optimized light field displays is that theavailable degrees of freedom may more flexibly be used to represent adesired light field under varying viewing conditions and tailored tospecific display hardware and scene content. However, the benefits ofoptimized light field displays may be best realized when there exists a1-to-many mapping between image element states and output ray states.Displays that constrain image element states in a 1-to-1 relationshipwith ray states impose a preconceived and unvarying mapping betweenimage elements and output rays, which cannot be adapted to varyingscenes and viewing conditions. This leads to low resolutionrepresentations of light fields, and in some cases the loss of imagebrightness. We call this 1-to-many mapping entanglement of the imageelement states and ray states. It should also be appreciated that anonlinear mathematical mapping between pixel states and ray space canexist even in the case of linear optical interactions. Nonlinear opticsprovide a means of creating nonlinear interaction, but not an exclusivemeans.

As described herein, an optimized light field display may be any displaythat generates content obtained by solving at least one optimizationproblem (e.g., using an iterative optimization algorithm or any othertype of optimization algorithm). In some embodiments, when an image isdesired from the display, an optimization problem may be posed, giventhe current state of the display, current state of the viewer, andcurrent knowledge of the desired display appearance, which optimizationproblem, when solved either explicitly or implicitly, by a computer orother means, will result in a display state that causes the display tooutput an image, which image may be an optimal approximation of thedesired display appearance. In this case an image is often a 4D lightfield, but does not have to be. (The desired output image can be a 3Dlight field, 2D image, 5D light field, vision correcting light field,accommodation cue light field, or many other desired display functions).

Optimized displays may employ the real-time or on-line content-basedoptimization techniques described herein. For pre-recorded images thatwill be viewed under predictable circumstances, it is possible for theoptimization problem to be posed in advance, and the solution to theoptimization problem may be generated (e.g., computing by solving theoptimization problem using an iterative gradient-based or otheroptimization algorithm) and stored for later retrieval and display.Because the output of such displays is also the result of anoptimization algorithm we consider displays that function in this way tobe optimized displays. In contrast, many lay-people use the term“optimized” to mean “tuned” or “adjusted” by some human-in-the-loop oropen-loop method. For example, a technician might be said to “optimize”the gamma value of a television display for a customer, when in practicethe technician is adjusting a parameter in a predetermined gammacorrection software module to a value referenced in a service manual.This does not mean that the television is an optimized display in thesense of the way in which this term is used herein, because there is nooptimization problem is solved to produce the output of the television.As another example, a display manufacturer might solve a formaloptimization problem to determine the values of a color lookup table, oreven the parameters of an algorithm, both for the purpose of convertinga 96-bit high-dynamic-range (HDR) image to a 16-bit HDR image to beshown on a HDR display. Such an HDR is not an optimized display in thesense of the way in which this term is used herein, because the outputof the display is not itself determined through formal optimization,even though an optimization technique was used to tune a function of thedisplay.

One compelling reason to use an optimized display, from a hardwaredesign perspective, is that the display gains flexibility of form andfunction with respect to traditional, fixed pipeline designs.Accordingly, in some embodiments, an optimized display may be treated asa system with a number of degrees of freedom, wherein the degrees offreedom can be applied, through optimization methods, to createsynthetic light fields with desired properties, such as highspatio-angular resolution, wide field of view, high display brightness,high temporal refresh rate, and good perceptual image quality (orfidelity). Moreover, a display driven by real-time optimization canadapt to changing viewing conditions as said viewing conditions change.Non-limiting examples of conditions to which the display may wish toadapt includes viewer position, ambient light intensity, ambient lightdirection, number of viewers changing display content (such as areal-time light field video stream), defects of the viewer's visualsystem, device power consumption requirements, device orientation, andviewer eye spacing.

How various factors in combination influence the quality of the imageshown on an optimized display is complex to predict. Another of the keybenefits of optimized displays described herein is that as desired thefactors that influence display quality can be traded-off against oneanother to maintain a desired level of display quality. Though each typeof display hardware will have its own set of factors that influencedisplay quality, the case of an optimized two-layer multiplicative lightfield display is typical of optimized displays. In the case of theoptimized two-layer multiplicative light field display the followingfactors may influence the displayed image quality for physical lightfield image¹: view disparity, layer positioning (e.g., the proximity ofa virtual object in the desired scene to the physical location of adisplay layer), scene brightness (e.g., how bright is the overall scenebeing displayed as a fraction of the maximum display brightness),computational time (e.g., the time available after rendering a scene todetermine the display layer patterns), and available power (e.g., theamount of device power available for computation and backlight).¹Non-physical light fields, which represent light ray paths inconsistentwith physical light propagation, have a related set of qualityinfluencing factors.

View disparity may be influenced by the field-of-view of the display(e.g., the viewing cone over which images are intended to be viewed),scene depth (e.g., the amount that objects in the scene extend into thedisplay away from the viewer or out of the display towards the viewer),and depth of field (DOF). The failure of a display to render the correctview disparity in physical scenes manifests as a spatial blur thatoccurs in regions of the scene that extend far from the plane of thescreen. This is known as DOF as the effect mimics the effect of the samename in camera systems. Though all automultiscopic displays have somedegree of DOF, optimized displays in accordance with some embodiments ofthe technology described here may achieve better DOF for a givenoperating point than conventional displays. Rendering views with acloser angular spacing is one way to increase the perceived quality ofthe DOF blur.

Following below are some illustrative and non-limiting examples of howan optimized display, implemented in accordance with the someembodiments of the technology described herein, may be configured todynamically trade between different features to support changing viewingconditions. In these examples it is typically assumed that the displayhas information about the location of viewers in front of the display,which can comprise head location information and eye locationinformation. In some embodiments, such tracking information may beobtained through a standard camera system, a stereo camera system, amulti-view camera system, or a depth camera system.

In one example, a user may take her device from a darkly lit room into adaylight environment. In some embodiments, the optimized display in thedevice may begin to operate closer to its maximum achievable brightnessin order to compete with the environmental lighting. As a result, thedegree of 3D pop-out may decreases slightly, and the displayed image mayexhibit some additional artifacts.

In another example, a first user may be viewing an immersive light fieldmovie on a tablet device with an optimized light field displayimplemented in accordance with some embodiments. A second user may sitsdown next to the first user to also view the movie. In some embodiments,the display may adapt to provide each viewer with unique, physicallycorrect, views of the light field. In order to achieve the extra degreesof freedom necessary to expand the field of view for both users, theimage may become slightly dimmer, the depth in the scene may becompressed, and the image quality may be slightly reduced.

In another example, a user may be viewing stereo content on an optimizedlight field display implemented in accordance with some embodiments. Asecond user sits down next to her to also view the movie. In order toconserve degrees of freedom and avoid the need to extrapolate from theprovided content, the display may replicate the same stereo views to theeyes of both viewers.

In another example, illustrated in FIG. 17, a viewer 1701 may beobserving a multi-view display 1704 implemented in accordance with someembodiments. In the pictured embodiment a single view frustum 1702 (alsodescribed as a view cone) denotes the angular region in which viewersmay view a 3D scene. In some embodiments of the technology describedherein, a tracking device 1703 may be used in order to track viewer1701, and adjust the view frustum 1702 in accordance with the spatiallocation of the viewer 1701 in relation to the multi-view display 1704,as measured by the tracker 1703. The view frustum 1702 may be adjustedto enclose both eyes of the viewer 1701. In this way, the display 1704may measure the eye positions of the viewer and adapt the size anddirection the output views to display only those views that arenecessary for the content to appear correct for the current viewer. Thiscan be done for multiple viewers. By displaying a view cone 1702 thatcovers a box just around the viewer's eyes 1701 the degrees of freedomof the display may be conserved for increased display brightness,extended depth-of-field, and other desirable properties available in theabove trade-space. The degree to which the display is able to target aviewer's eyes depends on the accuracy and latency of the tracker 1703employed to track the viewer's eyes. In applications with very lowtracker latency and very high tracker accuracy it is also possible todirect separate view cones to each of a viewer's two eyes.

In applications with higher tracker latency or lower tracker accuracy,it may be advantageous to expand the view cone around the viewer's eyes,such that tracker inaccuracy or update time does not produce anobservable visual defect. In other words, even if the tracker were offby a small amount, and the viewer moves before the tracker, renderer,and optimizer can update the optimal viewing location of the display,the viewer will still be within a valid, well defined, high-qualityviewing zone. If the primary source of inaccuracy is tracker latency,then the viewing cone should be expanded by a small amount in real-worldphysical units—for example the size of the view cone at the viewer's eyeplane should be 2 cm wider than the viewer's eye separation distance. Inthis situation the source of error (user movement) will not be affectedby the distance of the viewer from the display. If the primary source ofinaccuracy is tracker noise or misalignment, the view cone should beexpanded by a fixed angular increment, such as 1.5 degrees wider thanthe cone subtended by the viewer's eye box. In this case the source oferror is the tracker itself, and the error will be magnified inreal-world coordinates as the viewer moves away from the screen.

In another example, a mobile device may include an optimized displayimplemented in accordance with some embodiments. As the battery reservesof the mobile device are depleted, the display may be configured toconserve power at the expense of brightness and display quality. In someembodiments, the device may dim the backlight, while keeping the displayintensity the same at the cost of introducing image artifacts.Alternatively, the device may dim the overall intensity of the screen,or a combination of both. In some embodiments, the device may reduce theamount of processing done to solve the optimization problem, causing theimage quality to be reduced, or the device frame rate to be reduced, orthe convergence speed of the algorithm to be reduced. The latterapproach may be appropriate for content that rarely changes.

In another example, a monitor of a general purpose computer may includean optimized display implemented in accordance with some embodiments. Auser may run applications which present content on the display that isprimarily flat with small regions that pop-out to accentuate elements ofthe graphical user interface. In such instances, the display may run ina low-power mode, updating regions of the screen selectively, and usinga wide FOV so that the results displayed in the application can beshared among multiple viewers. As the user switches from desktopapplications to a game with dynamic graphics requiring larger 3D pop-outeffects (larger DOF), the display may narrow the FOV, and may increasethe display backlight brightness. This gives the display more degrees offreedom to represent light field content, at the expense of increasedpower consumption and a personal, single-viewer gaming experience.

In another example, an optimized light field display implemented inaccordance with some embodiments may be positioned in a hotel lobby togive patrons a glasses-free 3D map experience. When a single viewer islooking at the map, the experience is a bright, dynamic 3D scene with animpressive amount of 3D pop-out (large DOF). When a second patronapproaches the map, the display reduces the amount of 3D pop-out, andbecomes dimmer to allow for additional degrees of freedom required torepresent multiple view cones. As the display becomes surrounded byviewers it seamlessly falls back to a 2D viewing mode so that everyonecan clearly read the information on the screen.

V.A Far-Field Light Field Displays

Some embodiments described herein provide for far-field light fielddisplays. Some embodiments provide for a number of variations offar-field light field displays and are discussed herein.

Display Spatial Resolution

Conventional multi-layer displays are always composed of layers withequal spatial sampling. The inventors have appreciated that this is nota necessary condition for optimized multilayer light field displays. Tothe contrary, the inventors have identified many reasons to chooselayers with different spatial sampling patterns or frequencies. Forexample, in order to reduce optical Moire interference between the highfrequency sampling grids of two layered displays, it may beadvantageous, in some embodiments, to place an optical diffuser sheetbetween the layers. Many of the benefits of choosing different displayresolutions may be derived from achieving greater efficiency (e.g.,lower rendering costs, reduced power consumption) when confrontingoptical blur introduced intentionally by diffusers, or unintentionallyby diffraction or optical properties of display devices.

In some embodiments, in order to reduce the required strength of thediffuser used to attenuate Moire interference, the display samplespacing may be be adjusted to account for perspective projection from anexpected viewing location. If the viewer is expected to be distance dfrom the front layer of the display, and the displays are separated by adistance s, with the pixel spacing of the front display p_(ƒ), then thepixel spacing of the rear display may be determined as

$p_{r} = {p_{f} + {\frac{p_{f}s}{d}.}}$This works because moire is effectively the beat frequency between thesampling grids of the different layers as projected into the viewer'seye plane. In some embodiments, modification by the above amount maycause both screens to project to the same spatial frequency at thetargeted viewing distance, d. A weaker diffuser may be used to attenuatethe resulting Moire pattern for viewing distances close to d, primarilybecause small differences in spatial frequency at the projected eyeplane will result in low frequency Moire patterns, which are lessnoticeable to human viewers.

Alternatively, in some embodiments, displays with large differences inspatial frequency as projected to the viewer's eye will create highfrequency moire, which may be invisible or less visible to the viewer.Moire interference in a multiplicative space such as anattenuation-based multilayer display may lead to new perceivedfrequencies: the sum and difference of the spatial sampling frequenciesof the two panels as projected to the viewer's eye.

There are many possible advantageous configurations of displays havinglarge differences in spatial frequency. In one illustrative embodiment,two panels may be separated by a distance of 3 mm, one with a spatialresolution of 1000 dpi (25.4 micron pixel), and the other with a spatialresolution of 500 dpi (50.8 micron pixel). If a viewer is standing at 60cm directly in front of the screen, establishing an optical projectionthrough both screens to the viewer's eye, the pixels of the rear screen,projected onto the front screen will be 50.5 micron. This establishes aspatial sampling frequency of 39.4 pixels/mm and 19.8 pixels/mm for thefront and back screens, respectively. Moire patterns at 59.2 pixels/mmand 19.6 pixels/mm will be observed—both outside the range of visiblefrequencies for a viewer at 60 cm.

Additional benefits may be achieved by adjusting the color filters ofeach of the layers, in some embodiments. For example, removing the colorfilters of the front layer is equivalent to boosting the horizontalspatial sampling rate by a factor of 3, thus for appropriately chosenscreen sample rates and separation distances, reducing Moire intensityperceived by the viewer.

FIG. 21 shows a viewer 2106 observing a stack of optical modulators2101, 2103, and 2105 in accordance with some embodiments of thetechnology described herein. In some embodiments, the optical modulators2102, 2103 and 2015 comprise a grid structure, where said grid structureis proportional to the display resolution, which will create undesirableoptical interference called Moire interference. Placing a diffuserbetween two optical modulators can mitigate Moire interference. Theembodiment of FIG. 21 shows two optical diffusers 2102 and 2104.Diffusers 2102 and 2104 may have different diffusion weights, withdiffuser 2102 being a light weight diffuser and diffuser 2104 being aheavy weight diffuser. The diffusers 2102 and 2104 are placed atdifferent distances d_(a) and d_(b) in the optical stack. As a ray 2107passes through each of said diffusion layers 2102 and 2104, it will bescattered by a degree according to the weight of the diffuser. Saidscattering will impart an according amount of angular divergence, whichwill create optical blur as the ray travels a distance to the viewerposition 2106. The weight of diffuser 2104 may be chosen to mitigate theMoire interference between optical modulators 2103 and 2105. Dependingon the size of the grid structure of optical modulator layers 2101,2103, and 2105, additional diffusion may be required to mitigate theMoire interference between optical modulators 2101 and 2103, and may beadded by selecting a diffusion weight and distance for diffuser 2102. Insome embodiments, the display resolution of optical modulator 2101 willbe lower than that of 2103, and the display resolution of opticalmodulator 2103 may be lower than that of 2105. In some embodiments, thedisplay layer spacings s and t may differ, and in some embodiments theymay be identical.

Another benefit of reducing the spatial resolution of one or more layersof a display device that may be realized in combination with the abovenoted benefits is that of reduced rendering and optimizationcomputational costs. Optimization and rendering costs are function ofthe number of discrete rays it is possible to represent with a displaysystem. The number of discrete rays depends, combinatorially, on thenumber of image elements in each display layer. Often in the presence ofoptical blur, introduced intentionally to mitigate Moiré interference,or unintentionally through the properties of the various surroundinglayers in the display device, a single image element on the displaylayers furthest from the viewer will not be individually resolvable.Thus the computational cost to render and optimize content for thispixel grid will be wasted. In this case, it may be advantageous, in someembodiments, for the purpose of computational efficiency to reduce thenumber of image elements on said display layer until just before eachdisplay element will be individually resolvable by a viewer at theexpected viewing distance of the display.

Refresh Rate/Temporal Resolution

The optimization techniques described herein may be used to synthesizelight fields using many systems that conform to the nonlinearity andentanglement requirements laid out above. Additional degrees of freedom,provided by high temporal refresh rates allow for more flexibility inthe trade space described above. This means that optimized displays withhigh temporal refresh rates may produce higher quality images.

First it is helpful to discuss the variable flicker fusion rate of thehuman visual system, which rate is the rate beyond which a viewer willnot perceive flicker in foveal vision. The flicker fusion rate varieswith respect to absolute light intensity, and the relative magnitude ofthe flicker signal (difference between high intensity and lowintensity). Flicker fusion in bright lighting environments occurs atapproximately 60 Hz, and may occur at flicker rates as low as 40 Hz indarkened environments. In this discussion, a flicker fusion period willbe considered to be the reciprocal of the flicker fusion rate. In thecase of an optimized display where each layer in a stack of layerssimultaneously refreshes each pixel on said layer sequentially, suchthat each of said display layers simultaneously scans out one frame, theentire stack of layers will appear to have the same refresh rate as anyone element of the stack. Thus, a stack of layers that can scan out anentire frame in 1/240th of a second will have a refresh rate of 240 Hz.An optimized display constructed from this stack will have the abilityto scan out four images within the flicker fusion period of a viewer inbright lighting conditions, and six images within the flicker fusionperiod. Another way to look at flicker fusion is that the human visualsystem is temporally band limited and does not perceive out-of-bandtemporal frequencies.

One metric helpful in answering the question of refresh raterequirements for light field display is to consider the algebraic rankof the light field to be displayed as parameterized by the displaylayers, in comparison to the algebraic rank of the display output space.In the case of a 240 Hz 3 plane layered multiplicative displayconfiguration this would constitute considering the rank of the lightfield to be displayed (often between 10 and 20) parameterized as anorder-3 tensor, against an order-3, rank-4 tensor. For perfectreconstruction, the ranks would need to be equal. However, in practice,information is not uniformly distributed across all elements in arank-wise decomposition of the light field. This means that for somescenes, a rank-4 reconstruction of the light field may be of acceptablequality. A screen with a 1000 Hz refresh rate may be able to reconstructmost natural light field scenes using temporal multiplexing alone.

In order to achieve a benefit from high speed display layers, at leasttwo display layers in a stack should update at the same rate. A simplethought experiment reveals the reason that a single layer updatingfaster than other layers will not improve the degrees of freedom of theoptimized display. The human visual system will approximately averageover all frames that occur over a single flicker fusion period. If onlya single display layer has updated in one period, the problem becomeslinear, rather than multilinear, and the average of the frames displayedduring the flicker fusion period could be displayed on the rapidlyupdating layer at a lower rate.

This reasoning leads us to see that in a multi-layer system at least twolayers should be configured to update at the same rate if displayupdates are to be used efficiently. In some implementations, it mayadvantageous to hold at least one layer of a multi-layer optimizeddisplay constant while updating the remaining layers, so long as morethan one layer remains. This may be used to simplify computation,deliver results to the user faster (as opposed to spending more timecomputing the result), and to conserve power that would otherwise havegone to solving the optimization problem.

Modern display hardware updates each image element in sequence, oftencalled a raster scan, or scan-out. The sequence typically used begins atthe top row of the screen, and updates each pixel along the first linesequentially, then proceeds to the next line, continuing until theentire screen has been updated. To date, optimized displays have beenmade using commercial off-the-shelf display hardware which uses thisraster scan update sequence on each layer of the optimized display.

The inventors have recognized and appreciated, however, that it may beadvantageous, in some embodiments, to take a different, non-sequentialapproach to updating optical elements in display layers (e.g., whenusing commodity display hardware to create an optimized display). In anoptimized display, the inventors have recognized that it may beadvantageous to update the screen at a higher rate in order to increasethe number of degrees of freedom available for representing a desiredimage. In practice not all regions of a desired image will be equallycomplex to represent. Therefore, in practice, it may be beneficial toupdate some regions of the display layers quickly at the cost ofupdating other regions of the display layers more slowly. Non-sequentialupdates allow for this trade-off in update rate.

Modern display buses such as DisplayPort have bandwidth beyond therequirements of a standard display. In a sequential display updatescheme, no pixel addresses are required because it is assumed that theordering of the pixels determines their locations. Because of thebandwidth of modern display buses it is possible to pass a data blockcontaining pixel values and pixel coordinates (addresses) to a displaycontroller, allowing for arbitrary pixels to be updated.

On the other hand, in an optimized multilayer display, with displaylayers that use a non-sequential update scheme, regions of the displaycorresponding to high scene disparity or very bright imagery, orotherwise requiring many degrees of freedom to represent may be updatedmore rapidly, and darker, low disparity regions, or regions requiringfewer degrees of freedom to represent can be updated more slowly. Forexample, a 1024×768 pixel screen that is updated at 60 Hz, neglectingblanking intervals, each pixel is updated in 1/47185920 second (47 MHz).While some display technologies such as LCD have settling times thatwould not allow such rapid updates to a single image element, groups ofimage elements can still be updated at a rate much higher than 60 Hz.Conversely, in many display technologies a pixel will maintain its statefor longer than 1/60th of a second without being updated, meaning it ispossible to update other regions of the screen at lower rates withoutlosing the image data on the display.

Accordingly, in some embodiments, when using this non-sequentialaddressing scheme with an optimized multilayer display, it is possiblefor the optimization algorithm to direct the sequence of updates,pushing reconstruction error out of band, or above the flicker fusionrate of a human observer.

It should be appreciated that as the problem of light field display on amultiplicative multilayered display device is rank-1 in the sense thatthe light field emitted from a two layer screen is the outer product ofthe attenuation function displayed on each layer, so too is the problemof display scanout through standard row and column drivers a rank-1problem in the sense that the 2D image shown on the display is a rank-1outer product of the electrical states of the row and column drivercircuits. While the problem of addressing pixels with row and columndrivers is typically solved trivially today by selecting a single pixelto update at each time step, it is possible to repurpose displayrow-column driver circuits currently used in many display devices toinstead create a sequence of rank-1 images at each time step. A singleoptimization algorithm, of the general form outlined earlier in thisdocument, can be formulated to address a multitude of display layers,each of said layers being addressed by a row and column driver, withsaid drivers being sent actuation signals producing rank-1 images oneach of said display layers at each time step, and said multitude ofdisplay layers comprising an optimized light field display, the outputof said light field display being a rank-1 outer product of the imageson each display layer at each time step. Having many time steps whichoccur at a high frequency, it is possible to push error out of bandtemporally and make high quality images for a human viewer.

Some hybrid display layer update schemes compatible with existingdisplay controller firmware are also possible, in some embodiments. Forexample, it is possible to change the refresh rate of the displays inorder to save power when displaying low-complexity scenes. For example,scene content that changes very little over time, or scene content thathas low view disparity, will require lower display refresh rates to showwith high quality.

Display Layer Spacing

FIG. 18A shows a detail view of a pinhole display, which helps toillustrate some embodiments of the technology described herein.Modulators 1801 a and 1801 b comprise a pinhole display 1801. Individualpixels (1802, 1803, 1804) of modulator 1801 b may be observed throughtransparent pixel 1805 of modulator 1801 a. This creates three distinctview cones. FIG. 18B shows a more distant view of the arrangement inFIG. 18A, with corresponding view cones. A virtual object 1806 is shownrelative to the position of the display 1801. FIG. 18C depicts adifferent system at the same level of detail as FIG. 18B, which systemis configured for use with non-negative matrix factorization methodsknown in the art. Virtual object 1806 is shown relative to the positionsof modulators 1801 a and 1801 b, positioned further apart than in FIGS.18A and 18B for use with non-negative matrix factorization (NNMF).

In optimized multilayer light field displays, the choice of layerspacing may have a significant impact on the observed quality ofdisplayed images. In practice the features exploited by the optimizationalgorithm driving the optimized multilayer display, in combination withthe content to be displayed, will dictate the best strategy to be usedto achieve desired results.

In configurations wherein the optimization algorithm exploits the 3Dgeometry of a multilayer device to create light field effects, such asthose displays that use non-negative matrix factorization, the bestpractice is to space the layers at 30% to 40% of the distance of themaximum depth of the virtual scene to be represented. For example, if anoptimized display device using NNMF methods were deployed in anapplication where it were required to display light fields representing3D scenes with a real-world depth of 4 cm, the layer spacing should bebetween 1.2 cm and 1.6 cm. The display will more efficiently representobjects behind the central plane of the screen than in front of it.Therefore the 4 cm range of the display will be biased towards the backof the display. This rule of thumb provides a starting point, and theoptimal value will depend on many factors, including the FOV of thesystem, the content displayed on the system, the amount of blur that istolerable, the spatial resolution of the display layers, and thediffraction blur that is created by the sample grid of the front-mostpanel.

Another factor that is an important consideration for some types ofmultilayer systems is the depth location of the plane of maximumresolution. The plane with maximum spatial sampling in an optimizedmulti-layer display will occur on the front-most layer of the display,with high spatial sampling also available on other display layers of theoptimized display. Accordingly, in some embodiments, the display layersmay be placed at critical depths where, for example, legible text, is toappear. Insight about display performance may be gained by running asimulation of the physical system in conjunction with the optimizationframework and intended content.

In embodiments where an optimized display has hardware capable of hightemporal update rates (e.g., greater than 60 Hz), an optimizationalgorithm that exploits both the 3D geometry of a multilayer device andthe high temporal sample rate of the display may be used. In this casethe spacing guidelines may be relaxed, as the high speed temporalvariation of the display provides degrees of freedom that may be used bya suitable optimization algorithm to extend the DOF of the optimizeddisplay. The degree of relaxation depends significantly on the speed ofthe modulator, and may be understood through simulation.

In embodiments where an optimized display has hardware capable of highfrequency spatial sampling (e.g., a pixel grid having a pixel pitch ofless than 1 arc-minute at a typical viewing distance), an optimizationalgorithm that exploits both the 3D geometry of a multilayer device andthe high spatial sample rate of the display may be used. In these cases,the actuation patterns generating parallax-based effects may have asufficiently small pitch that the spacing between layers can also besmall (e.g., less than 6 mm). This makes the required layer spacing asnarrow or narrower than that required by a pinhole display.

Color

Conventional multi-layer optimized displays are generally created usingidentical LCD panels having color filter arrays, with each of the colorchannels being optimized independently from one another. The inventorshave developed techniques for jointly determine the actuation signalsfor multiple color channels of an optimized multi-layer display. Thisdevelopment allows for many new physical designs, and improvedperformance within existing hardware designs.

In optimized multi-layer displays comprising stacks of LCD panels whereeach LCD panel contains identical arrangements of identical color filterarrays, there will be cross-talk between the color filters, such thatthe color channels of the LCD are not truly isolated. In a hypotheticalmulti-layer display comprising a stack of LCD panels with ideal colorfilters, light transmitted through a red color filter in the first layerwould not be transmitted through a green filter of the second layer, andso on for each non-identical pair of color filters contained in thecolor filter arrays of the LCD panels. This idealized multi-layerdisplay system would allow for independent optimization of each colorchannel, as the color channels would not interact. However, in reality,cross-talk between the color filters will allow light from one colorchannel on one layer to pass through another color channel on anotherlayer. The inventors have appreciated that in order to accuratelyreproduce the desired colors in a displayed light field or image theactuation signals that drive the optimized display must be generatedjointly among all color channels and with knowledge of the cross-talkfunction between the color channels.

The inventors have recognized and appreciated that, in the case of anoptimized multi-layer display with identical color filters on eachlayer, where the actuation signals driving the optimized multi-layerdisplay are obtained jointly among all color channels, it is not onlypossible to obtain color-accurate representations of a desired lightfield or image, it is also possible to exploit the color cross-talk toachieve a wider color gamut, and spatially sub-sampled imagery. Thecolors of light that are emitted from pairs of non-identical colorfilters, one on each layer, will introduce new color primaries that canbe used for image formation. For instance, light emitted along a paththat intersects a red color filter on the first layer and a green colorfilter on the second layer might appear yellow-orange, depending on howthe product of the response curves of the color filters.

In an idealized multi-layer display in which color filter responses donot have cross-talk, and the color filters are arranged per-pixel intocolor sub-pixels, the spatial resolution of each color channel is thesame as the spatial resolution of the display. In this scenario, whenconsidering each color channel independently, only light rays thatintersect identical color filters on each display layer will propagate.This means that for each color channel the sample rate of the visiblecolor sub-pixels is the same as the overall display sample rate.However, in real-world displays exhibiting color cross-talk betweencolor filters, light will be visible along ray paths that intersectnon-identical color filters on different display layers, meaning thatfor at least a subset of colors the spatial sampling may be increased tothat of the sub-pixel sample rate.

At the time of display manufacture, color filter arrays may be tuned tomaximize the above noted effects of increased color gamut and increasedspatial sampling, as well as the optical intensity of the display.Generally, the goal of these modifications is to take advantage ofmultiple modulations that occur, one on each of the multiple displaylayers, to reduce the light loss through the color filters, and toincrease the variety of color primaries available at the output of thedisplay through combinations of filters on each of the layers. Becausethe observed color filter response for a ray exiting a multi-layereddisplay is the product of the color filter responses along the ray paththrough the stack of layers, multiplicative combinations of color filterresponses may be achieved.

FIGS. 19A and 19B depict color filter responses that may be used in, orexemplify the color filter responses used in colored pixels of opticalmodulators, in accordance with some embodiments of the technologydescribed herein. In embodiments where a display is created from twolight attenuating layers, the observed color along a single ray paththrough the layers will be the product of the color responses of eachlayer at the location where said ray intersects said layer. In FIG. 19A,solid lines 1901 and 1902 represent standard red and green color filterresponses found in LCD panels known in the art, respectively. Suchresponses might have a full-width-half-max of 70 nm. The dotted lines1903, 1904, and 1905 represent the product of the three combinations ofthe two color filters, respectively—1901 by 1901, 1901 by 1902, and 1902by 1902. The aforementioned products will comprise the possible colorsobserved by layering two layers comprising pixels with color responses1901 and 1902.

FIG. 19B depicts modified color responses shown as solid lines 1906 and1907. Dotted lines 1908, 1909, and 1910 denote the products of the threecombinations of the two color filters, respectively—1906 by 1906, 1906by 1907, and 1907 by 1907. Responses 1906 and 1907 have been modifiedfrom 1901 and 1902 such that their products have greater area under thecurve, thus allowing more light energy to pass through the colorfilters. The modified color filter responses will typically have afull-width-half-max of greater than 100 nm.

In some embodiments, the color filter responses may be modified bytaking the square root of a standard color filter response. It is alsopossible to determine the optimal color filter responses for a corpus oflight field images by running an optimization problem that minimizes thelight field reconstruction error within the corpus across a variable setof color filter responses. In some embodiments, color primaries may beremoved entirely from one or more layers of an optimized multi-layerlight field display. It is well known that chrominance is perceived at alower resolution than luminance by the human visual system. Thisperceptual fact can be exploited to create optimized multi-layer lightfield displays with better light efficiency by excluding color filterarrays from one or more layers in a multi-layer display.

In some embodiments, it may be possible to decompose the light fieldimages to be displayed by an optimized light field display intoluminance and chrominance components, then decimate the resolution ofthe chrominance component, and create a cost function for said optimizeddisplay that jointly incorporates the target high resolution luminanceand low resolution chrominance components. In embodiments where anoptical stack comprises one layer with an RGB color filter array as istypically found in LCD panels, and a second grayscale LCD without colorfilter arrays, each sub-pixel of the color filter array will contributeto the luminance image, while the lower resolution chrominance imagewill be formed between the color pixels of the color layer and thegrayscale pixels of the layer without color filter arrays.

Number of Display Layers

One of the ways to increase the number of degrees of freedom availableto an optimized light field display is to increase the number of layersin the display. There are inherent tradeoffs when increasing the numberof display layers in a multilayer display. While the extra degrees offreedom afforded by additional layers contribute to improving thetheoretical upper bound for display quality, in practice the addition ofa non-ideal display layer will increase light loss due to absorption andscattering, and create internal reflections that, if uncharacterized,will contribute to image artifacts.

It is possible to compensate for internal reflections by accounting forthe reflected ray paths in the formulation (e.g., via a suitable costfunction) of the optimization driving an optimized multi-layer display.Often it may be necessary to compensate for only the first bounce, butmulti-bounce corrections are also possible.

FIG. 20 illustrates techniques for compensating for internal reflectionswithin a multi-layer multi-view display, in accordance with someembodiments of the technology described herein. As shown, opticalmodulators 2001 and 2002 are placed a distance s apart to form a lightfield display. Ray 2005 is transmitted directly through both layers. Insome treatments ray 2006, which is emitted from x0_1, reflected fromx1_1 and x0_0 and passes through x1_0 before being observed by viewer2004 may be ignored. However, it may be advantageous to account for theeffect of ray 2006 and other such reflected rays. Two ray bundles aredrawn in FIG. 20. Ray bundle 2007 originates at optical modulator 2001and passes through optical modulator 2002, and ray bundle 2008,originates at optical modulator 2001, then bounces off of opticalmodulator 2002, then bounces off of optical modulator 2001, then passesthrough optical modulator 2002. By tracing the paths of ray bundles 2007and 2008 it is shown that the bouncing ray paths of ray bundle 2008 canequivalently be modeled as an additional additive optical modulatorlayer 2003 at a position 2 s from optical modulator 2001.

Moiré interference is caused by spatial aliasing between regularspatially distributed patterns, and is well understood in the opticscommunity. It has been shown that placing a diffuser between displaylayers in a multi-layer display can mitigate the effects of moiré byblurring out the high spatial frequencies that cause Moiré interference.Here, we show an alternative method for placing diffusers in multi-layerdisplays.

There are various issues associated with diffuser placement inmulti-layer displays, including those where the display has more thantwo layers. The general goal of the diffuser placement is to ensure thatthe resulting blur kernel from each of the layers of material in themulti-layer display stack, when combined, creates a blur kernel that isjust large enough to attenuate the spatial frequency of the displaysampling grid, from the perspective of the viewer, and is no larger thannecessary preventing the loss of observable spatial resolution. It isalso noted that the display layers themselves create a blur due todiffraction, and that this blur can be exploited to reduce Moiréinterference with layers closer to the viewer.

One method to find the amount of blur required between each layer is toproject the layer in question onto the next layer, multiply the signals,and compare the spectrum of the result to the minimum allowable Moiréfrequency, creating an aggregate characterization of the overall blurfunction. In an optimization-based multilayer display, the optimizationformulation would generally incorporate a characterization such thepreviously-mentioned characterization.

Optimized Display with Optical Modulators Optimized displays may beconstructed with a wide variety of light modulating devices such as,liquid crystal display devices, for example. In this section, weenumerate some useful combinations of devices that can comprise themodulator elements for an optimized light field display implemented inaccordance with some embodiments described herein. The modulatorcombinations outlined below may be used with many of the techniquesdescribed herein.

In some embodiments, a wide variety of LCD technologies may be usedincluding, but not limited to, TFT, TN, Pi cells, and ferroelectricLCDs.

In some embodiments, a light emitting device (e.g., an organic LEDdisplay) may be combined with an attenuating device (e.g., an LCDpanel), so long as there is at least one light attenuating device infront of at least one light emitting device. In some embodiments, anoptimized display may comprise an OLED display in a two layer stack asmall distance behind an LCD display. This has the advantage of reducedpower usage and increased update rate as compared to many LCD andbacklight combinations.

Micro-electro-mechanical systems have been used to create slidingshutter “light valves” which can switch between binary opacity states ata high rate. In some embodiments, these light valves may be employed,along with optimization methods that create binary patterns, to createan optimized light field display. In some embodiments, two MEMS shutterdevices may be arranged in a stack of layers. The spacing of the layersin these devices may be determined in any of the ways described herein.

An LCD panel may be manufactured such that the precision of themodulation state is traded for the number of discrete levels ofmodulation. It is therefore possible to create LCD panels, where theoutput of said panels is quantized to only a few bits. Typical LCDpanels have their output quantized to 6-bits, whereas typical imagesintended for display on LCD panels have at least eight bits of dynamicrange. Temporal and spatial dithering may be used to hide the differencebetween the desired output dynamic range and the available dynamic rangefrom a human viewer. In an optimized display comprising multiplicativelayers, the output dynamic range will be the sum of the dynamic range ofeach of the layers. For example, an optimized multilayer displaycomprising two 6-bit LCD layers will have a dynamic range of 12 bits.This may be accounted for in the optimization algorithm to increase thedynamic range of the light field emitted from an optimized multi-layerdisplay. The visual quality of displays with 8-bit output may also beimproved using the above strategy.

LCD panels are typically designed to be of appropriate thickness tofully modulate the polarization state of an incoming light wave.Reducing the thickness of the glass layers of an LCD device will reducethe amount of light that is modulated by the panel, reducing theobserved contrast of displayed images, but will also increase theswitching rate of the panel. The inventors have recognized that in amulti-layer optimized display, as it is possible to achieve increasedcontrast in displayed images through multiple modulations of a light raypassing through the multiple layers, and as it is possible to exploitincreased temporal bandwidth to improve said optimized display, making atrade-off between contrast and display update rate is advantageous.Typical LCD contrast is above 1:1000. In the case of an optimizedmulti-layer display it is possible to use LCDs with contrast of 1:100 orlower.

In some embodiments, optimized displays may be made from printedtransparency sheets. A novel aspect of the technology describe herein isthe thinness of the stack of passive modulating layers used to createthe perception of a 3D effect. When setting the actuation signals fromthe solution to optimization algorithms that utilize the spatial bandlimit of the human visual system, it is possible to make the layerspacing much thinner than reported in the literature, and it is possibleto use fewer layers than reported in the literature.

Reflection-Based Optimized Displays

Unlike conventional optimized displays, which are transmissive, theinventors have developed some optimized display architectures that allowfor reflective optimized light field displays. The reflection-modeoptimized displays described herein have two basic ingredients: amodulator, and a reflective back-plane. Though there is not a directcorrespondence to a transmissive optimized display, the analog of thereflective plane is the backlight unit and the modulators are the same.

FIGS. 22A-F illustrate aspects of reflection-mode multi-view displays,in accordance with some embodiments of the technology described herein.Depicted displays comprise at least one transmissive modulator and areflective backplane. Said backplane may be a diffuser reflector or aspecular reflector, and may itself also be a modulator. In thisdiscussion, a specular reflection may occur when an incident light ray,upon being reflected, remains compact enough in exit direction that itmeaningfully interacts with only a single modulator element on its exitpath from the system.

FIG. 22A shows a transmissive optical modulator 2203 in front of aspecular reflector 2204 (which for example may be a mirror). Lightoriginates at a distant point light source 2201, which produces parallelor nearly parallel rays. A viewer 2202 observes the display. Light raysreaching the viewer 2202 must travel through modulator 2203 twice—onepass through the modulator occurs on entry as the ray travels from lightsource 2201 towards specular reflector 2204, and the second pass throughthe modulator occurs on exit reflecting from specular reflector 2204towards the viewer 2202. Actuation signals may be produced to controlthe pixels of modulator 2203 in accordance with the methods describedherein to produce a multi-view display, and with knowledge of thelighting direction to light source 2201.

FIG. 22B shows another embodiment in which multiple transmissivemodulators 2207 and 2208 are placed in front of a specular reflector2209. In this embodiment, rays from distant point light source 2205 aremodulated four times—two modulations occur as the ray travels from lightsource 2205 to reflector 2209 and two additional modulations occur asthe light ray travels from specular reflector 2209 to viewer location2206. The embodiment of FIG. 22B allows for additional degrees offreedom in the system, which allows for improved multi-view displayperformance, including but not limited to expanded field of view, andincreased apparent pop-out. Actuation signals may be produced to controlthe pixels of modulators 2207 and 2208 in accordance with the techniquesdescribed herein to produce a multi-view display, and with knowledge ofthe lighting direction to light source 2205.

FIG. 22C shows an arrangement of spectral reflector 2214 andtransmissive modulator 2213. The point light source 2210 is placed closeto the display, such that rays arriving from light source 2210 cannot beconsidered to be parallel. This may lead to an undesirable situation inwhich regions of the display will appear dark to viewer 2212, as therewill be no ray path connecting the viewer position 2212 to the lightsource 2210 over some regions of the display. In some embodiments, theneed for a nearby light source can be accommodated by replacing pointlight source 2210 with area light source 2211. Knowledge of the positionand shape of the light source may be used to compute the actuationsignals for modulator 2213 in accordance with the techniques describedherein.

FIG. 22D depicts a transmissive modulator 2217 on top of a specularreflective modulator 2218. In this case, a distant point light source2215 will cast a ray that travels through modulator 2217, is modulatedupon reflection at reflective modulator 2218, and is modulated a thirdtime at modulator 2217 as it continues towards viewer location 2216. Insome embodiments, the arrangement of elements pictured in FIG. 22D willadd degrees of freedom to a reflective multi-view display system beyondthose that would be available with a specular reflective backplane aspictured in FIG. 22A. Actuation signals for modulators 2218 and 2217 maybe computed according to the techniques described herein, and withknowledge of the direction of incident light from light source 2215.

FIG. 22E depicts a transmissive modulator 2221 layered atop a diffusereflector 2222. The diffuse reflector scatters light such that a singleray from distant point light source 2219 is modulated by the modulatinglayer 2221, scattered by the diffuse reflector 2222, and then interactswith multiple pixels on the modulating layer 2221. A diffuse reflectorwith a wide reflection lobe, when coupled with a single distant pointlight source 2219, will form an image of the pattern displayed on themodulator on the diffuse reflector. Thus, from the perspective of theviewer, the pattern on the modulator 2221 modulator will interact with ascaled and shifted version of the same pattern, projected by the lightsource 2219 onto the diffuse reflector 2222. In some embodiments,actuation signals may be computed, according to the techniques describedherein, that incorporate said interaction between shifted patterns, inaddition to the viewer location 2220 to produce a multi-view display.

FIG. 22F depicts a configuration of transmissive modulator 2225 atop adiffuse reflector 2226, wherein the diffuse reflector has a reflectionlobe that is narrow. Said narrow lobe causes light reflected from saiddiffuse reflector to interact with a small subset of pixels on modulator2225. In some embodiments, a narrow diffuse reflection lobe, whichcauses light to be scattered in across an angular range that encompassesonly a subset of the pixels on the modulator 2225, enables a beneficialway of manage the degrees of freedom in a multi-view display system.Actuation signals may be computed for the system pictured in FIG. 22Faccording to the disclosed techniques. In some embodiments, it may beuseful to also consider the size of the diffuse lobe of the diffusereflector, the direction of the point light source 2223, and theposition of the viewer 2224.

With some types of modulating layers, it may be possible to achievegreater optical efficiency by making adjustments to the optical system.For example, in the case of an LCD panel in front of a reflectivesurface that preserves the polarization state of light, it may bepossible to remove the rear polarizing layer of the LCD in order toincrease the optical transmission. In this setup, it may be advantageousin some circumstances to replace the front polarizing layer of an LCDwith a circular polarizer to prevent the intensity distribution of thedisplay from being inverted optically. Coupling the above modified LCDmodulator with a reflective modulator capable of altering thepolarization state of incident light, such as a liquid crystal onsilicon (LCoS) device, has the effect of adding a reflective modulatorto the system while preserving the optical efficiency of a systemwithout a reflective modulator.

Backlight Design

The inventors have appreciated that, for efficiency purposes, thebacklight coupled to an optimized light field display should be designedto provide a directional light cone over the area of expected viewingonly. For example, if a display is expected to support viewers over a 90degree horizontal span, the backlight unit placed behind the modulatinglayers of the display should be made to emit light only across the 90degree span of expected viewing. This remains true in the case of adisplay that tracks one or more viewers' positions and updates theangular region over which the display shows content. In many cases thedisplay will have a narrow field of view at any moment, but the narrowfield of view can be aimed in a variety of directions such that thenarrow field of view may address a wider angular region of expectedviewing locations. In some embodiments, for these cases, in order tokeep the design of the backlight relatively simple, the backlight may bemade to emit light over the entire angular region of expected viewinglocations at once. In practice, due to the non-ideal nature of opticalmaterials it is expected that at least 90% of the light emitted by theefficient angular backlight falls in the angular region containing theexpected viewer locations.

V.B Near Eye Light Field Displays

Non-limiting examples of near eye light field displays include virtualreality displays (e.g., two LCD layers near the viewer's eye with a mainlens between the eye and the screen) and augmented reality displays(e.g., a virtual reality system having a split optical path). Virtualreality displays may be implemented using any of the multi-layerapproaches described herein. Augmented reality displays may beimplemented by including a second low-resolution multi-layer system toproduce occlusions.

FIG. 23 illustrates one embodiment of a multi-layer light field displaydesigned for the purpose of providing a light field input into the eyeof a viewer 2312, and designed to be located near to the viewer's eye.This arrangement is in contrast to other embodiments in which themodulating layers are far from the viewer's eye and the output isintended to present different imagery to multiple viewer eye locationssimultaneously. In the case of a near-eye light field display, someembodiments are designed to create multiple scene viewpoints across theviewer's pupil, allowing the viewer to refocus his or her eye on virtualobjects. A pair of such near-eye light field displays is able to createa virtual reality or augmented reality headset capable of providingaccommodation and focus cues to a viewer.

The system illustrated in FIG. 23 depicts one embodiment of thetechnology described herein comprising two optical paths: one opticalpath directs an image of the world, including objects or people in frontof the viewer into the viewer's eye, and a second optical path images aplurality of layered optical modulators into the viewer's eye to createa light field image of a virtual scene. The optical path to create thevirtual image begins at the top of FIG. 23 with backlight element 2301.A lens or lenses 2302 may be used to condition the backlightappropriately for the modulators, said conditioning comprising forexample, focusing or collimating the light source. An alternativeconfiguration for some embodiments would be to forego an emissivebacklight 2301 in favor of collecting environmental light using lens(es)2302 as environmental light collector(s). Optical modulators 2303 a and2303 b may optionally be high resolution modulators in some embodiments,and are shown by way of example as two layers of liquid crystal lightmodulating elements in FIG. 23. Layers 2303 a and 2303 b may be anytransmissive or reflective optical modulating elements. Layers 2303 aand 2303 b may be driven by computed actuation signals according to thetechnology disclosed herein. Lens(es) 2304 may be present to act asoptical adapters. Said optical adapters may couple the output from themodulators and from the optical combiner to adjacent optical elements.An optical path to image real-world scene images to the viewers eye 2312begins at the world 2305, which may be construed to represent any sceneto be viewed through the device. 2306 shows an optional adjustableoptical attenuator, which may be used to adapt the intensity of theworld 2305 to the intensity of light transmitted by the opticalmodulators 2303. Lens or lenses 2307 represent an optional main lens orscene collection optics, which may image the world 2305 into the system.Optical combiner 2308 may represent, for example, a pair of prisms. Saidoptical combiner will combine the images formed by main lens 2307 andlight field display 2303, wherein the combined images then travel in thedirection of the viewer 2312. Optical modulators 2309 a and 2309 b maybe present to create a dark field screen. Said screen is drawn, by wayof example and not limitation, as two liquid crystal light modulatingelements, but may comprise other types of optical modulators. Saidscreen is designed such that diffraction blur through the device isminimized at the viewer's eye location. In some embodiments, diffractionmay be minimized by creating the device from liquid crystal elementswith large apertures. The intention of dark field screen element is toblock light arriving from specified directions. Said dark field screenmay be known as an angularly dependent attenuator. In some embodiments,said dark field screen allows the viewer to observe correct or plausiblereal-world occlusion from virtual objects, or to remove real objectsfrom view. In some embodiments, the modulators 2309 a and 2309 b may bedriven using actuation signals computed according to the methodsdisclosed herein, given the appropriate cost function describing thedesired attenuation distribution. Element 2310 shows an optional relayoptic that can be used in some embodiments to adapt the system tospace-constrained designs. Lens(es) 2311 represent eyepiece optics,meant to couple the output of the system to the human visual system,indicated by eye location 2312.

FIGS. 24A and 24B illustrate embodiments of multi-view displays that maybe used for the application of near-eye light field display includingaugmented and virtual reality, in accordance with some embodiments ofthe technology described herein. Both figures depict embodiments ofnear-eye light field displays that make use of alternative opticalelements from those depicted in FIG. 23. The drawings of FIGS. 24A and24B are simplified to clarify the role of the featured optical elements,and are not intended to suggest that the full generality of the systemdrawn in FIG. 23 does not apply to the systems of FIGS. 24A and 24B.

In FIG. 24A, the world 2401 is imaged through an optical combiner 2402,which combiner combines the image of the world with the light fieldimage produced by modulators 2403 a and 2403 b and directs said imagesto the viewer 2405. Modulators 2403 a and 2403 b are, in someembodiments, driven by actuation signals produced according to thetechnology disclosed herein to produce a multi-view display. Reflectiveoptics 2404 deliver the image produced by modulators 2403 a and 2403 bto the viewer 2405 via the optical combiner 2402.

In FIG. 24B, an embodiment is illustrated using wedge optics 2408 tocombine images of the world 2406 and light field images from modulators2407 a and 2407 b. Said images are directed to the viewer 2409.Modulators 2407 a and 2407 b may be actuated by actuation signalscomputed in accordance with the technology disclosed herein. Embodimentsof the disclosed technology using wedge optics may benefit from spaceefficient designs.

V.C Projected Augmented Reality Light Field Displays

A projected, augmented reality display may be made using the principlesoutlined in this document, in conjunction with a projection lens andviewing surface. The purpose of such a display would be to projectvirtual images into views of the world as seen through a window,windshield, cockpit, view-screen, projection screen, or otherreflective, transparent surface, such that the projected virtual imagesintegrate with the real scene, in 3D, from the perspective of one ormore observers.

It has been shown that optimized displays can project light field imageswith specialized screen optics. The inventors have developed techniquesfor creating an optimized projected light field display for augmentedreality applications without using a specialized screen. The basiclayout of such a device will comprise a source, which providesspatio-angular variation, a main lens, which scales the source to thedesired spatial scale, and a screen, which reflects the projected imagesto the eye of the viewer.

In some embodiments, the source may comprise any one of the transmissiveoptimized light field displays detailed in the sections above, coupledwith appropriate backlighting. In some embodiments, the screen may beany reflective or semi-reflective surface, optionally modified to makeit preferentially reflective from the known location of the source andmain lens, in the known direction of the viewer. Such modification canbe achieved through holographic elements, or hogels, which are commonlyused for such purposes.

In some embodiments, surfaces for the screen may include carwindshields, boat and airplane cockpits, and exterior windows inbuildings and other structures. In the case that the reflective surfacehas a curvature, the light field emitted from the source may bepre-warped in order to compensate for the shape of the reflector. Thisis done by tracing the ray path from the light field source, through alloptical surfaces in the system, to possible locations for the viewer'seyes. These ray paths are then sorted to provide the desired shape ofthe light field. The key to creating a projected augmented realitydisplay is to ensure sufficient angular diversity reaches the viewinglocations of the viewer's eyes. In order to achieve this, the anglesubtended by the exit pupil of the main lens of the system must be atleast as large as the desired field-of-view of the system. For thisreason, a large lens placed near to the viewer is desirable, in order toachieve a system with a workable field-of-view. Large lenses willnecessitate large modulating layers in the source. These canpreferentially be achieved with LCD devices. In order to reduce the costof the main lens, it is sometimes desirable to use Fresnel optics, orcatadioptric systems (those which combine mirrored devices andrefractive optics) in the construction of the main lens. Multiple typesof modulators can be used, including LCD, LCoS, MEMS DMD, MEMS Shutterdisplays.

V.D Calibration of Optimized Light Field Displays

In some embodiments, a mobile device may be used to calibrate algorithmsused herein for determining actuation signals for optimized displays.For example, a mobile device may be used to aid in determining themappings from the various actuation signals to the display viewsCalibration methods in a manufacturing environment would also follow ina straightforward way from these examples and considerations.

In some embodiments, calibration using a mobile device may include anysuitable combination of one or more of the following: (1) the use of amobile device camera, depth camera, or camera array in imaging thedisplay being calibrated; (2) the use of a camera, depth camera, orcamera array attached to the display being calibrated, in imaging themobile device; (3) the use of the display being calibrated in displayingvarious calibration patterns, including QR codes, to be viewed by ahuman or by a mobile device camera, depth camera, or camera array; (4)the use of the mobile device display in displaying various calibrationpatterns, including QR codes, to be viewed by a human or by a cameraattached to the display being calibrated; (5) the use of multiple mobiledevices in performing calibration; (6) the continual adaptation ofvarious displayed patterns as calibration parameters are learned; (7)the use of accelerometer, gyroscope, compass, sonar or othermeasurements by the mobile device in determining the position of themobile device with respect to the optimized display; and (8) the use ofwireless or wired data transmission in exchanging parameters between themobile device and computation attached to the display being calibrated.

FIG. 25 shows an illustrative example of using a mobile device forcalibrating a multi-view display, in accordance with some embodiments ofthe technology described herein. In embodiments of the technologydescribed herein that comprise multiple light modulating layers therelative spatial orientation of said layers will affect the multi-viewimagery produced by the display. Using methods known to those skilled inthe art display layers may be arranged in predetermined positions andsaid positions characterized at the time of manufacture. In someembodiments, it may be necessary to re-characterize the positions of themodulator layers relative to one another, for example because ofmechanical changes in the device, errors in manufacture, and/or softwarechanges.

FIG. 25 illustrates an example of a system to perform saidcharacterization via a calibration step in-the-field using a smartphoneor other mobile computing device. In some embodiments, calibrationin-the-field may refer to any calibration performed post-manufacture. Insome embodiments, the system shown in FIG. 25 (and its variations) maybe applied at the time of manufacture.

In the embodiment of FIG. 25, a multi-layer display to be calibrated2501 comprises two modulating layers displaying calibration patterns2503 and 2504 and a camera 2506. The spatial relationship between thecamera and at least one modulating layer may or may not be known. Alsoillustrated is a mobile device 2502 comprising a display, which is ableto display a calibration pattern 2507 visible to camera 2506, and acamera 2505, which is able to image calibration patterns 2503 and 2504.Both cameras 2505 and 2506 may be integrated cameras, camera arrays, ordepth cameras. In some embodiments, the spatial relationship betweencamera 2505 and pattern 2507 is known. In some embodiments a wirelesscommunication channel 2508 is present between mobile device 2502 andscreen 2501.

In some embodiments of the calibration procedure, mobile device 2502 ismoved to a plurality of locations in front of display 2501. The numberof locations may be greater than a threshold number (e.g., greater than3, 5, 10, 20, 25, etc.). At each of said locations camera 2505 is ableto image patterns 2503 and 2504, and camera 2506 is able to imagepattern 2507. Using camera calibration techniques (e.g., as may be usedin computer vision), the spatial position of patterns 2503 and 2504 areestablished with respect to camera 2505, and the spatial position ofcalibration pattern 2507 is established with respect to camera 2506. Insome embodiments, the relative arrangement of patterns 2503 and 2504from the perspective of camera 2505 may be sufficient for calibration.In some embodiments, using the known spatial relationship between camera2505 and pattern 2507, the coordinate systems of cameras 2505 and 2506may be connected, establishing a global coordinate system for allcalibration patterns. From the established spatial location of eachpattern the spatial orientation of each modulator may be derived. Insome embodiments, the spatial information is shared between the mobiledevice 2502 and screen 2501 via (e.g., wireless) communication link2508. In some embodiments, the communication link 2508 comprises aconnection via a server on the Internet or other network.

In FIG. 25 the calibration pattern is drawn as a QR code as anon-limiting example. The calibration pattern may comprise other popularpatterns including, for example, the chess board pattern or dotcalibration pattern in the OpenCV software package or any other suitablepattern. In some embodiments, the calibration patterns may includeunique metadata. In some embodiments, unique metadata and calibrationdata obtained during the calibration may be stored for later retrievalin a database, and said database may reside in a server on the Internetor other network. The server may be a cloud server. Calibration patterns2503 and 2504 are drawn such that they do not overlap, even though themodulators in screen 2501 are placed on top of one another. In someembodiments, the calibration patterns may not overlap in the view ofcamera 2505. In some embodiments, overlapping calibration patterns maybe used to achieve additional measurement accuracy. Overlappingcalibration patterns may produce a Moire effect, which can be measuredby camera 2505 to determine the separation between modulator layers.

V.E Caching and Compositing of Computed Actuation Patterns

In determining the values of display actuation signals in optimizedmulti-layer displays, including using the previously-mentionedtechniques employing blurring transformations, the inventors haverecognized the advantages of caching of actuation signals and/or thecompositing of cached signals with other cached signals as well as thosesignals under optimization.

In some embodiments, generated actuation signals may be stored. Thegenerated actuation signals may be obtained in any of the ways describedherein including as the result of an explicit optimization process orusing a heuristic method for generation. The generated actuation signalsmay be generated for: (1) a pre-selected view position or set of viewpositions; (2) in the context of an animated scene, a pre-selectedanimation frame or set of animation frames; (3) in the context of scenesdisplaying content as a function of various parameter values, apre-selected parameter value or set of parameter values, and/or in anyother suitable context. The actuation signals may be stored in any typeof memory, as aspects of the technology described herein are not limitedin this respect.

In some embodiments, stored actuation signals may be displayed(recalled) on a single- or multi-layer display, including an optimizeddisplay.

In some embodiments, the stored actuation signals may be combined one ormore other signals including, but not limited to: (1) actuation signalscorresponding to two-dimensional, diffuse light fields, e.g.two-dimensional text, images, graphics, or user interface elements; (2)other previously-recalled actuation signals; (3) actuation signals beingoptimized, explicitly or heuristically, in “real time”; and (4) anyother suitable actuation signals.

In some embodiments, the techniques for combining actuation signalsinclude: (1) spatial compositing based on individual display layers; (2)gradual blurring between edges of actuation signals being composited;(3) higher-order light field based compositing (e.g., compositing thatutilizes the known location of the viewer to identify partitions of theactuation signals that correspond to mutually distinct portions of thelight field generated by the display, and compositing that utilizes theknown location of the viewer to identify partitions of the actuationsignals that correspond to mutually distinct portions of perceived viewsof the generated light field); (4) additive or linear compositing, e.g.in a multiplicative multi-layer display where, given two sets ofactuation signals to be composited (“source signals”), all but one ofthe display layers have identical source signals. In this case thecomposited signal would correspond to a linear superposition of thesource signals on the display layer having distinct source signals; and(5) the use of a cached actuation signal or signals as parameters in, orinitial state of, an optimization algorithm that in turn generatesactuation patterns.

V.F Applications

It should be appreciated that aspects of the technology described hereinmay be used in a variety of applications. Non-limiting examples of suchapplications are automotive applications (e.g., 3D Dashboard, gauges,instruments, buttons, and augmented reality windshields), userinterfaces for computing devices (e.g., tablet, phone, laptop, desktop,workstation, etc.), computer-aided design (CAD) workstations,architectural models, entertainment applications (e.g., television,theater movies, home movies, mobile device movies), gaming applications(livingroom scale, handheld, desktop computing, public venue—arcade,team sports arena), medical imaging (e.g., visualizing volumetric dataobtained for example by an MRI or ultrasound imaging device, visualizingdepth data (2.5D), visualizing rendered/mesh data), scientific andmedical visualization applications (e.g., underground imaging,biological systems imaging, big data/data trends), financialapplications (e.g., complex markets, finding correlations), advertisingapplications, art applications, remote vehicle control applications(e.g., generating user interfaces for controlling unmanned vehicles inair, water, and/or land), monitoring single or multiple vehicles,teleconferencing applications, telepresence applications, teleoperationapplications, near-eye display applications (e.g., augmented realityand/or virtual reality applications), and creating accommodation cuesfor any of these applications.

VI. Further Descriptions of Techniques for Manufacturing Light FieldPrints

Further aspects of techniques for rapid, robust, and precisemanufacturing of light field prints are described in this section.

FIG. 26 shows an illustrative system 2600 for generating patterns to beprinted on layers of a light field print and printing the generatedpatterns on the layers of the light field print, in accordance with someembodiments of the technology described herein. In FIG. 26, linesindicate the path of data through the system, and storage indicatesparts of said data path where data may be stored. In some embodiments,the storage locations may be bypassed.

The input into the system pictured in FIG. 26 may comprise one of anumber of formats. In one embodiment, input 2601 may comprise aplurality of 2D views of a 3D scene, in some cases referred to as alight field. In some embodiments, input 2601 comprises a scenedescription including but not limited to geometry or textureinformation. In embodiments in which a scene description comprises theinput, the input may be converted 2602 to a light field representation,comprising a plurality images representing views of the described scene2603. When the input is already a plurality of images representing sceneviews 2604, the conversion step 2602 may be bypassed. In block 2606, thedesired light field representation 2605 may be used to compute thetarget patterns 2607 for printing onto layers to be assembled into alight field print.

In some embodiments, geometry, color model, and resolution information2608 may be incorporated into the computation of the patterns 2607. Insome embodiments, the target patterns 2607 may be processed at act 2609to compensate for properties of the print process. Such properties mayinclude, for example, physical properties of the medium, physicalproperties of the print process, dynamic properties of the printprocess, and fluid dynamic properties of the printer ink, or physicalproperties of printer toner. In some embodiments, processing 2609incorporates a physical model of the properties to be compensated 2610.In some embodiments, computation blocks 2609 and 2602 may be combinedinto unified computational system or method 2612. The compensatedpatterns 2611 may be sent to a printer or print spooler or otherwisereproduced in print.

In some embodiments computation block 2602 generates a representation ofa light field from a scene description, which scene description maycomprise a 3D scene represented, for example, as a CAD file, a depthmap, the OBJ file format, Collada file format 3D Studio file format,three.js JSON file format, or scene meant for ray tracing such as aPOV-Ray scene or Nvidia Optix program. The resulting desired light field2603 may be generated using any of numerous rendering techniques. Forexample, said rendering techniques may comprise a virtual multi-camerarendering rig to render an plurality of off-axis images from differentperspectives, GPU shader-based rendering techniques, and/or ray-tracingtechniques.

The light field generated by 2602 may be encoded in various formats. Forexample, the light field may be encoded as an ensemble of imagescorresponding to various desired views of the scene. In thisrepresentation, each pixel value corresponds to the desired color and/orintensity of a light ray to be emitted from a specific location and at aspecific angle on the display surface. The importance of the particularlight ray may also be encoded. In some embodiments, said encoding may beused to weight the error function used in the downstream processing2606.

Several methods may be used for computing target patterns for printing2606. In some embodiments, target patterns are computed for one printedlayer that is monochrome and a second printer layer that is color. Insome embodiments, target patterns that are binary in each ink channelmay be computed. For example, the patterns may comprise binary Cyan,binary Magenta, binary Yellow, binary blacK (CMYK) channels. Similarconsiderations may also be made for other color combinations and inksets, including without limitation light inks such as light black, lightcyan, and light magenta, spot color inks, and inks intended to extendthe color gamut of the printer. The computation of binary patterns maybe done, for example, by introducing appropriate regularization into thecomputational methods used to compute the target patterns in accordancewith the techniques described herein (e.g., using the techniquesdescribed in FIG. 2-13) disclosed herein. In some embodiments, patternsmay be computed by operating on sub-blocks of the target patterns, andcombining said sub-blocks to obtain the target patterns. In someembodiments, said sub-blocks may use associated partitions of the targetlight field. For example, the block processing may be done on eachiteration of any iterative method for performing the computation.

Some embodiments may include techniques for compensate for print andmedium dynamics in printing patterns for printed multi-view displays.The goal of said compensation is to obtain a compensated pattern from atarget pattern, where said compensated pattern has been corrected forprint and medium dynamics. For example, the compensated pattern may becorrected for any one or more (e.g., all) of ink bleed, dot gain, andthe maximum allowable ink density of the print medium.

Techniques for compensating for dot gain in creating light field printsinclude, for example, linear spatial filtering of the target patternsuch as Gaussian blur filtering, followed by an intensity thresholdoperation and/or the use of morphological processing methods. Prior toemploying these techniques, the target patterns may be spatiallyupsampled. Dot gain compensation methods used in creating light fieldprints may be applied to individual color channels or to multiplechannels jointly. The output patterns generated by dot gain compensationprocessing may be referred to as intermediate patterns.

Techniques for ink density compensation in creating light field printsinclude, but are not limited to, applying a structured pattern to theintermediate patterns, whereby a select number of individual pixels areeliminated so that ink, toner, dye, or other media, is not deposited onthe medium at the locations of the eliminated pixels. In someembodiments, the choice of which pixels to eliminate may depend on thepatterns upstream in the processing. In other embodiments, that choicemay be independent of the patterns upstream in the processing. Theresult of ink density compensation, in some embodiments obtained byprocessing the intermediate patterns, becomes the compensated patternsutilized downstream in the printing.

FIGS. 27A and 27B shows an illustrative example of a light field print,manufactured in accordance with some embodiments of the technologydescribed herein. The light field print in FIG. 27A comprises a frontprinted layer 2701 sitting directly atop, so as to be in contact with, arear printed layer 2702. The layers are lit by a backlight unitcomprising lamps, including without limitation LEDs 2703 and a lightguide 2704.

FIG. 27B illustrates a light field print comprising a front printedlayer 2705, separated from a rear printed layer 2707 by a transparentspacer 2706. This embodiment can also be illuminated by a backlightidentical in construction to that illustrated in FIG. 27A, comprisinglamps 2708 and light guide 2709.

In some embodiments, the ink or emulsion may be on the front-facingsurface of the printed layers 2701, 2702, 2705 and 2707. In someembodiments, the ink or emulsion may be on the rear-facing surface ofthe layer. In some embodiments, the entire layer may be a selectivelytransparent attenuator throughout the volume of the layer. In someembodiments, the particular mode of attenuation (e.g., top, bottom, orvolumetric) may be distinct between layers 2701 and 2702. By choosing anappropriate mode of attenuation, the thickness of the transparentmaterial onto which the pattern is printed may be used as a transparentspacer.

LEDs 2703 and light guide 2704 illustrate a side-illuminated backlightmodule. Alternative types of backlight modules may be used in otherembodiments. The backlight modules may be based on electroluminescent,fluorescent or LED elements, organized in side-illuminating,front-illuminating, or rear-illuminating configurations. The sameconsiderations apply to 2708, 2709.

FIG. 28 shows another illustrative example of a light field print,manufactured in accordance with some embodiments of the technologydescribed herein. Said light field print comprises a stack of emissiveand attenuating layers. Said layers may correspond to associated methodsdisclosed herein for sequentially assembling the individual layers toform a stack of printed layers. A printed pattern 2802 is printed ontothe surface of a backlight, comprising a lamp 2803 and light guide 2804.Illumination source 2803 may be, by way of example and not limitation,an LED. Frustrated total internal reflection results in the appearanceof an illuminated region at any location where ink is deposited 2802 onthe surface of the backlight medium 2804. An attenuating layer 2801 isthen affixed to the rear emissive layer. In some embodiments,attenuating layer 2801 may be affixed directly to emissive layer 2804.In some embodiments, attenuating layer 2801 is separated by a spacinglayer. The actuation signals for target and compensated patterns onlayer 2801 and ink layer 2802 may be computed according to techniquesdescribed herein for computing actuation signals for prints containing aplurality of attenuating layers.

FIG. 29 shows an illustrative example of a light field printmanufactured using a self-aligned printing method, in accordance withsome embodiments of the technology described herein. Illustrated are atransparent layer 2904 onto which a rear pattern 2903 is printed. Atransparent separator 2902 is affixed atop the printed pattern 2903. Insome embodiments, the separator may be affixed using an opticaladhesive. A front pattern 2901 is then printed on transparent separator2902. In some embodiments, spacer 2902 is affixed without influencingthe spatial location of transparent layer 2904. This may includeperforming assembly directly on the print bed or paten, and performingrepeated print passes for multiple layers. In some embodiments, a UVcured flat-bed inkjet printer may be used. In this way, the alignmentbetween layers and between each layer and the print head may bepreserved. In some embodiments the stack of materials 2901-2904 may beplaced on a backlight comprising edge-lit illumination source 2905 andlight guide 2906.

FIG. 30 shows an illustrative system for adaptively aligning theprinting process used for printing a layer of a light field print, inaccordance with some embodiments of the technology described herein. Aprint carriage 3001 moves along a guiding structure 3002. In someembodiments, a guiding structure may be used to move print carriage 3001in a direction normal to the drawing plane. In some embodiments, theprint medium is moved normal to the drawing plane to allow the printmechanism to address a two dimensional region of the print medium.

FIG. 30 illustrates a two-pass printing process. During the first pass,ink is deposited on a print medium 3004 by ink head 3007, which mediumis resting on print bed 3006. The ink may be aqueous ink, pigment inkdye ink, UV cured ink, and/or any other suitable type of ink. In someembodiments, the medium 3004 is a transparent layer that may be a rigidor flexible. The stack of material may be secured to the print bed 3006.In some embodiments, the material may be secured to the print bed byusing an arrangement of pins and/or a vacuum table.

After completion of the first pass the print medium 3004 is flippedover. The material stack 3003-3005 rests on the print bed or platen3006, and may be secured to the print bed as above. During the secondpass, the ink head 3007 deposits ink drops 3003 onto the medium 3004.Cameras 3008-3009 integrated into the print carriage 3001 allow formonitoring the second pass of the print process. The spatialrelationship of the plurality of cameras (drawn as 3008-3009) and theink head 3007 may be known. Camera 3003 monitors the ink currentlydeposited, viewed through the material stack 3003-3005. Camera 3008monitors the layer of previously-deposited ink 3005 through transparentlayer 3004, and any element of the stack below 3005. In someembodiments, the platen or print bed 3006 may incorporate anillumination source. In some embodiments, the platen or print bed 3006may incorporate tracking markers. As the print carriage 3001 moves overthe print bed or platen 3006, the cameras 3008-3009 are used to alignthe position of the print carriage 3001 to the previously-deposited inkpattern 3005. This helps to ensures that the print layers are aligned.

Light field prints manufactured in accordance with embodiments describedherein may be used in any suitable application. For example, light fieldprints manufactured in accordance with the techniques described hereinmay be used to create signage (e.g., advertising signage that, forexample, might appear in backlit light boxes in airports, malls, busstops and/or any other suitable locations). Signage may either have anintegrated backlight, or it may be installed as a rigid or flexiblemulti-layer print inserted into an existing back light box. As anotherexample, light field prints manufactured in accordance with thetechniques described herein may be used to create souvenirs and/orkeepsakes (e.g., a light field print of a person's face created from a3D scan of the person's head, sports memorabilia, architectural and/orany other suitable type of memorabilia). As another example, light fieldprints manufactured in accordance with the techniques described hereinmay be used to create logos, promotional materials, instructionalmaterials, and/or any other suitable type of materials used in abusiness environment.

As yet another example, in some embodiments, light field printsmanufactured in accordance with the techniques described herein may beused as glass décor (e.g., window décor), interior decoration, interiorbranding and/or artwork.

FIG. 31 shows a description of print service, indicating its use inadvertising signage. In some embodiments of the print service, a 3Dmodel is stored on a computer 3101. The 3D model may comprise a scenedescription, which may be represented, for example, as a CAD file, adepth map, the OBJ file format, Collada file format 3D Studio fileformat, three.js JSON file format, or scene meant for ray tracing suchas a POV-Ray scene or Nvidia Optix program. In some embodiments, the 3Dmodel comprises a plurality of 2D layers, with said 2D layer to be shownat one or more depths in a 3D scene. In some embodiments, 2D fileformats, which include without limitation Adobe Photoshop files, AdobeIllustrator files, bitmap images, vector images, may be upconvertedthrough a 3D upconversion process to create 3D models. In someembodiments, 2D files containing layer information may be upconverted to3D models by separating existing layers in depth.

In some embodiments, a 3D model may submitted to the print service 3102.The submission may comprise transmitting a file via the Internet. Insome embodiments 3D, model submission may take place via a web service,whereby 3D models submitted to the web service may be stored on a remoteserver. In some embodiments, 3D models may be created in a web browserand submitted to the print service 3102 via transmission over theInternet. In some embodiments, 3D models may be combined and edited oncesubmitted to the print service. In some embodiments, 3D models may besubmitted to the print service from a mobile device (e.g., a tablet,laptop, smart phone, appliance, etc.).

In some embodiments, the print service may allow users to import 3Dmodels from partner web services. The importing may include selectingone or more models from a partner web service, and communicating saidmodels or selections to the print service. The communicating maycomprise using of an API or a partner API. In some embodiments, theprint service may provide database of stock content from which users maycreate 3D scenes to be printed via the print service. The stock contentprovided by the print service may be free or may be available by a feeor subscription. The stock content may be provided by other users of theprint service for free or for a fee. The stock content may be availableon a market on the print service, and provided by a third party. Themarket may be a fee, barter, or trade market.

In some embodiments, 3D models submitted to the print service may bepreviewed upon submission. Said preview may show the predicted outcomeof printing or rendering the submitted 3D model with the print service.When creating said preview, the following inputs, without limitation,may be considered: 3D model contents, printer type, printer qualitysettings, algorithm inputs, algorithm settings, ordered viewing angle,viewing location, environmental lighting, backlight unit type, printmedium type, print inks, available compute time, product type, paymenttype or plan, print lifetime, and print environmental exposure.

In some embodiments, the preview available in the print service may bepresented on a standard 2D screen. In other embodiments, the printservice may allow for preview using glasses-free 3D screens, stereo ormulti-view screens, glasses-based 3D screens, virtual reality headsets,augmented reality headsets, or augmented reality on a mobile device suchas a smartphone or tablet. In the case of said augmented realitypreviews, the preview may be physically located, said physical locationcomprising the intended location of a print to be produced with theprint service, or another similar or dissimilar location. In someembodiments the user of the print service may have the opportunity toapprove or reject the print, for example, based on cost estimatesprovided by the print service and/or previews of the final product.

In some embodiments, a user may initiate a printing process using theprint service. The initiation may involve a payment, clicking a buttonusing a browser, or providing verbal input to technician or by means ofa kiosk at a storefront.

Upon initiation, the print service may process the submitted 3D scenedescription or 3D model according to the methods disclosed herein. Saidprocessing may generate target patterns for printing using techniquesdescribed herein. The generated target patterns may be compensated toproduce a compensated patterns using techniques described herein. Thetarget patterns or compensated target patterns may be sent to a printer3103 (e.g., a conventional roll inkjet printer as pictured, or a UVflatbed printer, or any other type of process that can modify thetransparency of a transparent material). The printer may create multipleprinted layer from the target patterns or compensated target patternssent to the printer. The printed layers may be aligned after printing,though in some embodiments they may be aligned at print time. Thealigned printed layers may be placed in front of a light box 3104, suchas a light box used for commercial signage applications. Such lightboxes may be found at transit stations, malls, movie theaters, and manyother commercial locations, as well as private showrooms, and homes, andentertainment venues and museums, and/or any other suitable locations.In some embodiments, the print service places prints in front of lightboxes. In some embodiments, the print service coordinates with a partnerorganization to print and distribute prints to light boxes. In someembodiments, the print service may offer user tracking or face trackingdevices to be co-installed with a printed sign. The tracking device maymeasure user engagement.

In some embodiments the above service may comprise a digital displayservice rather than a printed display service. In such embodimentsactuation signals will be generated from submitted 3D scene descriptionsor 3D models. In some embodiments actuation signals will be transmittedto digital displays comprising multiple optical modulators. In someembodiments said transmission may occur over the internet. In someembodiments the digital displays may be coupled with a tracking deviceto measure quantities related to viewership, by way of example and notlimitation, user engagement, number of viewers, and type of viewers. Insome embodiments said measured quantities may be used to tailor thecontent of said digital display.

In some embodiments, a light field print may be realized as lightattenuating patterned material and may be attached to both sides of aglass surface (e.g., a window). The patterned material may include filmthat has been printed, a color layer directly applied on the glasssurface, and/or material that has been machined or molded to contain alight attenuating pattern. The patterns on the patterned material may beprinted according to how light rays should be attenuated when passingthrough the glass surface, depending on the point at which a ray passesand the angle of the ray.

For example, when the glass surface is a window, the window may blocksunlight at a certain angle (e.g., as shown in FIG. 32), and pass it atanother angle (e.g., as shown in FIG. 33). In some embodiments, thepatterns deposited on a window may be generated, according to thelatitude, longitude and/or orientation of the window, which correspondsto different sunlight angles at specific time of the day and year. FIG.32 illustrates blocking sun-light at a certain angle and shows lightattenuating layers 3201 and 3203, window surface 3202, and sun 3204.FIG. 33 illustrates passing a sun ray and shows light attenuating layers3301 and 3303, window surface 3302, and sun 3304.

In some embodiments, light attenuating patterned material may bemanufactured to block man made light, such as for selectivelyattenuating light of passing cars, streetlamps, advertisements orpreventing light leakage from inside a room. In some embodiments, lightattenuating patterned material may be manufactured to display 3Dcontent, such as graphics or logos, that would be part of interiorand/or exterior branding.

In some embodiments, light attenuating patterned material may blocklight rays in one direction, but may include colors that reflect inanother direction, thereby display 2D graphics when viewed from the sameside at which a light source is located.

FIG. 34 is a flowchart of an illustrative process 3400 for manufacturinga light field print, in accordance with some embodiments of thetechnology described herein. Process 3400 may be performed by anysuitable system including, for example, system 110 or system 2600.Process 3400 begins at act 3402, where a plurality of scene views may beobtained, which scene views are to be rendered using a light field printbeing manufactured via process 3400. Each of the plurality of sceneviews may correspond to a location of a viewer of the light field print.As described herein, the scene views may be of a natural or a syntheticscene. Each scene view may comprise a grayscale and/or a color image ofany suitable resolution for each of one or more (e.g., all) of the sceneviews. Any suitable number of scene views may be obtained at act 1502(e.g., at least two, at least ten, at least fifty, at least 100, atleast 500, between 2 and 1000, between 10 and 800, or in any othersuitable combination of these ranges), as aspects of the technologyprovided herein are not limited in this respect.

In some embodiments, the scene views may be obtained by accessing and/orreceiving one or more images from at least one image source (e.g.,accessing stored images, receiving images from another applicationprogram or remote computing device). In some embodiments, the sceneviews may be obtained by first obtaining a description of a 3D scene(e.g., a 3D model of a scene) and then generating, as part of process3400, the scene views based on the obtained description of the 3D scene.

Next, process 3400 proceeds to act 3404, where printing processinformation may be obtained. Printing process information may includeany of the information 116 described with reference to FIG. 1B and, forexample, may include layer geometry information, color modelinformation, print resolution information, and/or any information thatmay be used for compensating the target patterns for print dynamics(e.g., at act 3410). In some embodiments, layer geometry information mayinclude information describing the size, shape, and position of thelayers relative to one another in the light field print to be assembled.For example, layer geometry information may indicated that each of thelayers is a plane and 11 inches in width and 17 inches in height, andthat the layers may be spaced apart 0.045 inches in the light fieldprint to be assembled. As another example, the layers may be curvedshapes that are to be spaced apart at a displacement of 0.06 inchesrelative to the surface normal in the light field print to be assembled.Layer geometry information may be expressed as a geometric model in asoftware package (e.g., AUTOCAD) or as a file (e.g., an OBJ file).

In some embodiments, color model information may specify a color modelthat represents the color channels available (e.g., the available inkchannels and ink set in a set of print heads) and/or optical propertiesof an ink set (e.g., spectral properties, information about how colorsinteract with one another when ink of one color is overlaid on ink ofanother color). Additionally or alternatively, the color model mayinclude any information embedded in a printer profile (e.g., an ICCdevice profile), and may contain information about how to map the devicecolor space (e.g., in the language of PostScript, a DeviceN orDeviceCMYK space) to a standard color space (e.g., sRGB). The colormodel may describe the optical properties of ink colors, non-limitingexamples of which include cyan, magenta, yellow, black, light cyan,light magenta, orange, green, red, violet, light black, light lightblack, matte black, glossy black, clear inks, emissive inks, glossoptimizers, and specific standardized colors such as Pantone colors.

In some embodiments, print resolution information may include the numberof addressable dot centers per inch, both in the horizontal and verticaldimensions (e.g., horizontal and vertical DPI). Print resolutioninformation may, additionally or alternatively, include the dot pitch orselection of dot pitches (dot radius or selection of dot radii)producible by the printing system (e.g., measured in inches or fractionsthereof). An example dot pitch may be 1/800 inch.

Next, process 3400 proceeds to act 3406, where information specifying atleast one blurring transformation may be obtained. The informationspecifying the at least one blurring transformation may specify one ormultiple blurring transformations and may include information of anysuitable type including, for example, any of information 114 describedwith reference to FIG. 1B.

Next, process 3400 proceeds to act 3408, where a plurality of actuationsignals may be generated based on the plurality of scene views obtainedat act 3402, printing process information obtained at act 3404, andinformation specifying at least one blurring transformation obtained atact 3406. This may be done in any of the ways described herein and, forexample, by using any of the optimization techniques described withreference to FIGS. 2-13.

Next, process 3400 proceeds to act 3410, where the target patternsgenerated at act 3408 may be compensated for print and/or mediumdynamics to obtain compensated target patterns (e.g., compensated foreffects of dot gain, for effects of printing material bleed, and foreffects of maximum allowable printing material density). Thecompensation may be performed in any of the ways described herein or inany other suitable way.

Next, process 3400 proceeds to act 3412, where the compensated targetpatterns are printed on the front and back transparent layers using aprinter of any suitable type including any of the types described hereinor any other technique for depositing the compensated target patternsonto the layers. After the target patterns are printed onto the layers,the layers may be assembled at act 3414 to create the light field print.Assembling layers into a light field print may include, for example,aligning the prints and adhering them to one another (e.g., using anadhesive or any other suitable means). After act 3414, process 3400completes.

It should be appreciated that process 3400 is illustrative and thatthere are variations. For example, in some embodiments, one or more ofacts 3406 and/or 3410 may be omitted.

VII. Additional Implementation Detail

In the embodiment shown in FIG. 35, the computer 3500 includes aprocessing unit 3501 having one or more processors and a non-transitorycomputer-readable storage medium 3502 that may include, for example,volatile and/or non-volatile memory. The memory 3502 may store one ormore instructions to program the processing unit 3501 to perform any ofthe functions described herein. The computer 3500 may also include othertypes of non-transitory computer-readable medium, such as storage 3505(e.g., one or more disk drives) in addition to the system memory 3502.The storage 3505 may also store one or more application programs and/orresources used by application programs (e.g., software libraries), whichmay be loaded into the memory 3502.

The computer 3500 may have one or more input devices and/or outputdevices, such as devices 3506 and 3507 illustrated in FIG. 35. Thesedevices can be used, among other things, to present a user interface.Examples of output devices that can be used to provide a user interfaceinclude printers or display screens for visual presentation of outputand speakers or other sound generating devices for audible presentationof output. Examples of input devices that can be used for a userinterface include keyboards and pointing devices, such as mice, touchpads, and digitizing tablets. As another example, the input devices 3507may include a microphone for capturing audio signals, and the outputdevices 3506 may include a display screen for visually rendering, and/ora speaker for audibly rendering, recognized text.

As shown in FIG. 35, the computer 3500 may also comprise one or morenetwork interfaces (e.g., the network interface 3508) to enablecommunication via various networks (e.g., the network 3510). Examples ofnetworks include a local area network or a wide area network, such as anenterprise network or the Internet. Such networks may be based on anysuitable technology and may operate according to any suitable protocoland may include wireless networks, wired networks or fiber opticnetworks.

In some embodiments, an active display may comprise an arrangement ofactive electro-optical elements whose actuation signals dynamicallyspecify their various optical properties, for example light transmissionin the case of an LCD element stacked between front and rearpolarization layers.

In some embodiments, actuation signals may comprise a set ofcontrollable variables that determine the optical behavior of elementsin an optical stack. For example, pixel intensity values sent to aprinter would be actuation signals determining light attenuation on aprinted transparency; pixel (color) values sent to a color LCD panelwithout a backlight but including front and rear polarizers would beactuation signals determining light attenuation by the panel. Values ofactuation signals may be determined using any of the techniquesdescribed herein. In some instances, the value of an actuation signalmay be fixed.

In some embodiments, a light field may comprise the spatio-angulardistribution of light in an environment. For example, the surface of amulti-layer optical stack may have a well-defined light-field, as wouldthe natural environment surrounding a person.

In some embodiments, an optical stack may comprise a general arrangementof passive and active optical and electro-optical elements, including:LCDs, OLEDs, polarizers, wave retarders, backlights, other illuminationsources, color filters, lenses, lenslet arrays, parallax barriers,optical splitters, optical combiners, diffractive elements, holographicelements, and optical films.

In some embodiments, a parametric display may comprise to a multi-viewdisplay wherein the mapping between actuation signals and thespatio-angular intensity distribution emitted from the display isnon-linear (e.g., wherein the parameters are not readily interpretableas basis coefficients in a basis expansion of the spatio-angularintensity distribution).

In some embodiments, a passive display may refer to an arrangement ofpassive, fixed optical elements whose actuation signals specify theirvarious optical properties, for example light transmission in the caseof a printed transparent medium. A passive display may include an active(energy-consuming) or passive backlight.

In some embodiments, a multiplicative layer may comprise an opticalelement having a multiplicative effect on light intensity, whereintensity is multiplied by some general nonlinear function of theassociated actuation signal. For example, the non-linear function mayincorporate a gamma curve or other perceptual intensity curve.

In some embodiments, a modulating layer may comprise to an active orpassive optical element whose optical properties are controlled.

In some embodiments, a backlight may provide illumination for one ormore transmissive components of a display device. The transmissivecomponents may be optically closer to a viewer, compared to thebacklight, which may be optically further from the viewer.

In some embodiments, “intensity” may refer to any measure of or relatedto intensity, energy, or power. For example, the “intensity” of lightincludes any of the following measures: irradiance, spectral irradiance,radiant energy, radiant flux, spectral power, radiant intensity,spectral intensity, radiance, spectral radiance, radiant exitance,radiant emittance, spectral radiant exitance, spectral radiantemittance, radiosity, radiant exposure and radiant energy density.

Aspects of the technology described herein may have the followingconfigurations.

(1) An optimization-based multi-view display system, comprising two ormore modulating layers.

(2) A parametric, multi-view display system, comprising two or moremodulating layers.

(3) The system according to (1), wherein the viewer position is utilizedin determining the optimization problem.

(4) The system of according to (1), wherein the ambient light intensityis utilized in determining the optimization problem.

(5) The system of according to (1), wherein the ambient light directionis utilized in determining the optimization problem.

(6) The system according to (1) wherein the number of viewers isutilized in determining the optimization problem.

(7) The system according to (1) wherein the content to be shown on thedisplay is utilized in determining the optimization problem.

(8) The system according to (1) wherein the defects in one or moreviewers' visual systems is utilized in determining the optimizationproblem.

(9) The system according to (1) wherein the interocular spacing of theviewer is utilized in determining the optimization problem.

(10) The system according to (1) wherein the display is integrated intoa device, and wherein the power consumption requirement of said deviceis utilized in determining the optimization problem.

(11) The system according to (1) wherein the display is integrated intoa device, and wherein the device orientation with respect to a viewer orviewers is utilized in determining the optimization problem.

(12) The system according to (2) wherein the viewer position is utilizedin determining the parametric mapping.

(13) An optimization-based multi-view display system, comprising two ormore modulating layers wherein the spatial sample rate, i.e. the spatialsample density, of the modulation patterns on the layers differs by atleast one part in 1000.

(14) The system according to (13) wherein the spatial sample rate of thelayers is selected so as to adjust the frequency of moiré interferenceobserved by a viewer.

(15) The system according to (13) wherein the spatial sample rate of thelayers is selected so as to reduce the computational cost of renderingcontent for the display.

(16) The system according to (1) wherein the image elements of eachmodulating layer are updated non-sequentially.

(17) The system according to (16) wherein the order of updates of theimage elements of each modulating layer is the partial or final resultof an optimization problem.

(18) The system according to (1) wherein the update rate of the displaysis adjusted according to the spatio-temporal complexity of the contentto be displayed.

(19) The system according to (1) wherein the displayed multi-view imagesare interpreted by a human viewer to represent one or more virtualobjects that extend over a depth range in a region in front of thedisplay, behind the display, or both; and wherein the modulating layersare spaced in depth at a distance between 30% and 40% of the maximumdepth extent (i.e. the maximum achievable front-to-back scene depth, ofa virtual object or virtual objects shown on the display).(20) The system according to (1) wherein the displayed multi-view imagesare interpreted by a human viewer to represent one or more virtualobjects that extend over a depth range in a region in front of thedisplay, behind the display, or both; and wherein the modulating layersare spaced in depth at a distance less than 30% of the maximum depthextent, i.e. the maximum front-to-back scene depth, of a virtual objector virtual objects shown on the display.(21) The system according to (1) wherein the displayed multi-view imagesare interpreted by a human viewer to represent one or more virtualobjects that extend over a depth range in a region in front of thedisplay, behind the display, or both; and wherein the optimizationproblem utilizes the temporal band limit of the human visual system increating the actuation signals; and wherein the modulating layers arespaced in depth at a distance between 20% and 40% of the maximum depthextent, i.e. the maximum achievable front-to-back scene depth, of avirtual object or virtual objects shown on the display.(22) The system according to (1) wherein the displayed multi-view imagesare interpreted by a human viewer to represent one or more virtualobjects that extend over a depth range in a region in front of thedisplay, behind the display, or both; and wherein the optimizationproblem utilizes the temporal band limit of the human visual system increating the actuation signals; and wherein the modulating layers arespaced in depth at a distance less than 30% of the maximum depth extent,i.e. the maximum front-to-back scene depth, of a virtual object orvirtual objects shown on the display.(23) The system according to (1) wherein the modulating layers arespaced in depth at a distance less than 6 mm.(24) The system according to (23) wherein the optimization problemutilizes a band-limited perceptual model, e.g. the spatial band limit ofthe human visual system, in creating the actuation signals(25) The system according to (1) wherein there exists one or morepreferred depths in a virtual scene, to be displayed on an optimizeddisplay, for high resolution text and graphics to be displayed; andwherein one or more of the physical display layers is placed at one ormore of said preferred depths.(26) The system according to (1) wherein color images are shown by meansof attaching color filter arrays to the modulating elements of themodulating layers; and, wherein the color channels of the display areoptimized jointly.(27) The system according to (26) wherein a characterization of thecrosstalk between color filters in the color filter arrays is utilizedin the optimization algorithm.(28) The system according to (26) wherein a wider color gamut isobtained in the displayed images as compared to a single-layer display.(29) The system according to (26) wherein the color filters used in thecolor filter arrays are identical among the modulating layers.(30) The system according to (26) wherein the color filters used in thecolor filter arrays of each layer of the multi-layer display have afull-width half-max response of 70 nm or more.(31) The system according to (26) wherein the color filters used in thecolor filter arrays of each layer of the multi-layer display have afull-width half-max response of more than 100 nm.(32) The system according to (1) wherein color images are shown by meansof attaching color filter arrays to the modulating elements of only onemodulating layer.(33) The system according to (32) wherein the images to be shown aredecomposed into a luminance channel that is sampled at the same samplerate as the modulators comprising the display layers, and a chrominancechannel that is subsampled below the rate of said modulators.(34) The system according to (1) wherein the reflected light pathbetween the display layers is modeled in the optimization problem thatcreates the actuation signals.(35) The system according to (34) wherein the displayed multi-viewimages are interpreted by a human viewer to represent one or morevirtual objects that extend over a depth range in a region in front ofthe display, behind the display, or both; and wherein the reflectedlight path between layers is exploited to extend said depth range to adistance of at least 1.5 times the spacing between the layers.(36) An optimization-based multi-view display system, comprising threeor more modulating layers wherein diffusers are placed between thedisplay layers; and wherein said diffusers have weights that determine adegree of optical blur; and wherein said diffuser weights are selectedsuch that the degree of optical blur reduces the observed moiréinterference.(37) The system according to (1) wherein the optimization result isconstrained to binary values (fully opaque or fully transparent).(38) The system according to (37) wherein the modulating layers are madefrom LCD panels; and wherein the LCD panels are binary Pi cells.(39) The system according to (37) wherein the modulating layers are madefrom LCD panels; and wherein the LCD panels are binary ferroelectricliquid crystal arrays.(40) The system according to (37) wherein the modulating layers are madefrom MEMS shutter devices.(41) The system according to (37) wherein the modulating layers are madefrom MEMS mirror devices with accompanying optical elements.(42) The system according to (1) wherein the modulating elements of themodulating layers have fewer than 8-bits of precision (low bit depth);and wherein the final or intermediate solution to the optimizationproblem is quantized to the precision of the display devices.(43) The system according to (42) wherein the modulating elements of themodulating layers have fewer than 8-bits of precision (low bit depth);and wherein at least one such low bit depth modulating layer is an LCDpanel.(44) The system according to (1) wherein the contrast of at least one ofthe modulating layers is less than 1:100.(45) The system according to (44) wherein the layer with contrast below1:100 is made from an LCD.(46) The system according to (1) wherein at least one layer comprises anarray of light emitting diodes; and wherein at least one layer comprisesa light attenuating modulator; and wherein said layers comprising anarray of light emitting diodes is placed behind at least one of saidlayers comprising a light attenuating modulator, from the perspective ofa viewer.(47) The system according to (1) wherein at least one of the modulatinglayers comprises a passive, patterned light attenuating material.(48) The system according to (47) wherein said passive materialcomprises a transparent plastic sheet, patterned with ink.(49) The system according to (48) wherein the ink is deposited by aninkjet printer(50) The system according to (47) wherein said passive materialcomprises a transparent plastic sheet, patterned by toner, e.g. from alaser printer or other electrostatic printing method.(51) The system according to (47) wherein said passive materialcomprises a glass sheet patterned by ink.(52) The system according to (47) wherein said passive materialcomprises an optically-exposed film, e.g. a photomask.(53) The system according to (47) wherein said passive material has beenpatterned using a chromogenic process.(54) The system according to (1) wherein at least one of the modulatinglayers comprises a passive, machined layer.(55) The system according to (54) wherein the machined layer is madefrom wood.(56) The system according to (54) wherein the machined layer is madefrom metal.(57) The system according to (54) wherein the machined layer is madefrom opaque plastic.(58) The system according to (47) wherein the spacing between themodulating layers is less than 5 mm.(59) The system according to (47) wherein the spacing between themodulating layers is less than 60 times the width of the smallestfeature size of the modulating layer.(60) An optimization-based multi-view display system, comprising atleast one modulating layer, and at least one reflective layer.(61) The system according to (60) wherein the position of an incidentlight source is utilized in the optimization; and wherein the positionof the viewer is utilized in the optimization.(62) The system according to (61) wherein the reflective layer has adiffuse lobe.(63) The system according to (60) wherein the reflective layer is also amodulating layer.(64) The system according to (63) wherein the reflective layer comprisesan e-ink or e-paper display.(65) The system according to (61) wherein the reflective layer is alsotransmissive.(66) The system according to (65) wherein the reflective and transmissielayer comprises a transflective LCD.(67) The system according to (1) wherein the display is illuminated by abacklight unit; and wherein the backlight unit emits more than 90% ofthe total light emitted by said backlight unit over the angular regioncontaining the expected viewing locations of the display.(68) The system according to (1) wherein the system provides a 3D dashdisplay in a vehicle.(69) The system according to (1) wherein the system provides a 3Dinstrument cluster in a vehicle.(70) The system according to (1) wherein the system provides a 3Dcontrol surface in a vehicle.(71) The system according to (1) wherein the system provides a 3D userinterface on a smartphone.(72) The system according to (1) wherein the system provides a 3D userinterface on a tablet computer.(73) The system according to (1) wherein the system provides a 3D userinterface on a laptop computer.(74) The system according to (1) wherein the system provides a 3D userinterface on a desktop computer.(75) The system according to (1) wherein the system provides a 3D userinterface on a workstation computer.(76) The system according to (1) wherein the system provides a 3Dvisualization on a CAD workstation.(77) The system according to (1) wherein the system provides a 3D viewof an architectural model.(78) The system according to (1) wherein the system provides a 3D screenfor watching stereoscopic video content.(79) The system according to (1) wherein the system provides a 3D screenfor watching multi-view television or other multi-view video content.(80) The system according to (79) wherein the display is in a privatehome.(81) The system according to (79) wherein the display is in a movietheater.(82) The system according to (1) wherein the system provides a 3D gamingexperience.(83) The system according to (82) wherein the 3D gaming experience is ona mobile device.(84) The system according to (82) wherein the 3D gaming experience is ina private home.(85) The system according to (82) wherein the 3D gaming experience is ona desktop computer.(86) The system according to (82) wherein the 3D gaming experience is ona console gaming system.(87) The system according to (82) wherein the 3D gaming experience is ina public arcade.(88) The system according to (1) wherein the system provides a 3D viewof medical imaging data.(89) The system according to (1) wherein the system provides a 3D viewof volumetric medical scans.(90) The system according to (1) wherein the system provides a 3D viewmedical chart data or vital patient data.(91) The system according to (1) wherein the system provides a 3D viewof seismic imaging data.(92) The system according to (1) wherein the system provides a 3D viewof microscopic structures.(93) The system according to (1) wherein the system provides a 3D viewof trends in large data sets.(94) The system according to (1) wherein the system provides a 3D viewof financial data.(95) The system according to (1) wherein the system provides a 3D mediumfor artistic works.(96) The system according to (1) wherein the system provides a 3Dadvertisement.(97) The system according to (1) wherein the system provides a 3Dannouncement.(98) The system according to (1) wherein the system provides a 3Dvisualization for remote vehicle operation.(99) The system according to (98) wherein the vehicle is airborne.(100) The system according to (98) wherein the vehicle is submersible.(101) The system according to (98) wherein the vehicle travels overland.(102) The system according to (98) wherein the vehicle operates inspace.(103) The system according to (98) wherein the vehicle works onextraplanetary bodies.(104) The system according to (1) wherein the system provides a 3Dvisualization for controlling a fleet of autonomous vehicles.(105) The system according to (1) wherein the system provides 3Dteleconferencing.(106) The system according to (1) wherein the system provides 3Dtelepresence.(107) The system according to (1) wherein the system provides 3Dteleoperation.(108) The system according to (47) wherein the system provides a 3Dgraphic or text functioning as an indicator light.(109) The system according to (47) wherein the system provides a 3Dgraphic or text functioning as an emblem in a control panel for anelectronic system.(110) The system according to (47) wherein the system provides a 3Dgraphic or text functioning as a label for or on a switch, button,capacitive-touch button, knob, slide control, optical proximity sensoror other physical control element or surface.(111) The system according to (47) wherein the system provides a 3Dgraphic or text overlaid on an external light in a vehicle, e.g. a brakelight, tail light or other external illuminated surface.(112) The system according to (47) wherein the system provides a 3Dadvertisement.(113) The system according to (47) wherein the system provides a 3Dannouncement.(114) The system according to (47) wherein layers in the system areadhered to glass, e.g. a window.(115) The system according to (1) wherein the displayed multi-viewimages are interpreted by a human viewer to represent one or morevirtual objects; and wherein the said virtual objects appear to coexistwith true objects in the physical world as perceived by a human viewer;and wherein the virtual objects appear to have 3D shape.(116) The system according to (115) wherein the light emitted from thesystem is viewed by a viewer through a semi-reflective screen.(117) The system according to (116) wherein said semi-reflective screenis the windshield of a car.(118) The system according to (116) wherein said semi-reflective screenis the cockpit of an aircraft.(119) The system according to (116) wherein said semi-reflective screenis the window of a building or structure.(120) The system according to (116) wherein the light emitted from themodulating layers is transmitted through a lens.(121) The system according to (116) wherein the light emitted from themodulating layers is transmitted through a catadioptric system.(122) The system according to (116) wherein the light emitted from themodulating layers is transmitted through a system of curved mirrors.(123) The system according to (2) wherein the viewer position isutilized in determining the parametric mapping.(124) The system according to (2) wherein the ambient light intensity isutilized in determining the parametric mapping.(125) The system according to (2) wherein the ambient light direction isutilized in determining the parametric mapping.(126) The system according to (2) wherein the number of viewers isutilized in determining the parametric mapping.(127) The system according to (2) wherein the content to be shown on thedisplay is utilized in determining the parametric mapping.(128) The system according to (2) wherein the defects in one or moreviewers' visual systems is utilized in determining the parametricmapping.(129) The system according to (2) wherein the interocular spacing of theviewer is utilized in determining the parametric mapping.(130) The system according to (2) wherein the viewer position isutilized in determining the parametric mapping.(131) An parametric multi-view display system, comprising two or moremodulating layers wherein the spatial sample rate, i.e. the spatialsample density, of the modulation patterns on the layers differs by atleast one part in 1000.(132) The system according to (131) wherein the spatial sample rate ofthe layers is selected so as to adjust the frequency of moiréinterference observed by a viewer.(133) The system according to (131) wherein the spatial sample rate ofthe layers is selected so as to reduce the computational cost ofrendering content for the display.(134) The system according to (2) wherein the image elements of eachmodulating layer are updated non-sequentially.(135) The system according to (134) wherein the order of updates of theimage elements of each modulating layer is the partial or final resultof an optimization problem.(136) The system according to (2) wherein the update rate of thedisplays is adjusted according to the spatio-temporal complexity of thecontent to be displayed.(137) The system according to (2) wherein the displayed multi-viewimages are interpreted by a human viewer to represent one or morevirtual objects that extend over a depth range in a region in front ofthe display, behind the display, or both; and wherein the modulatinglayers are spaced in depth at a distance between 30% and 40% of themaximum depth extent, i.e. the maximum achievable front-to-back scenedepth, of a virtual object or virtual objects shown on the display.(138) The system according to (2) wherein the displayed multi-viewimages are interpreted by a human viewer to represent one or morevirtual objects that extend over a depth range in a region in front ofthe display, behind the display, or both; and wherein the modulatinglayers are spaced in depth at a distance less than 30% of the maximumdepth extent, i.e. the maximum front-to-back scene depth, of a virtualobject or virtual objects shown on the display.(139) The system according to (2) wherein the displayed multi-viewimages are interpreted by a human viewer to represent one or morevirtual objects that extend over a depth range in a region in front ofthe display, behind the display, or both; and wherein the optimizationproblem utilizes the temporal band limit of the human visual system increating the actuation signals; and wherein the modulating layers arespaced in depth at a distance between 20% and 40% of the maximum depthextent, i.e. the maximum achievable front-to-back scene depth, of avirtual object or virtual objects shown on the display.(140) The system according to (2) wherein the modulating layers arespaced in depth at a distance less than 6 mm.(141) The system according to (140) wherein the optimization problemutilizes a band-limited perceptual model, e.g. the spatial band limit ofthe human visual system, in creating the actuation signals.(142) The system according to (2) wherein there exists one or morepreferred depths in a virtual scene, to be displayed on a parametricdisplay, for high resolution text and graphics to be displayed; andwherein one or more of the physical display layers is placed at one ormore of said preferred depths.(143) The system according to (2) wherein color images are shown bymeans of attaching color filter arrays to the modulating elements of themodulating layers; and wherein the parameters representing colorchannels of the display are considered jointly.(144) The system according to (143) wherein a wider color gamut isobtained in the displayed images as compared to a single-layer display.(145) The system according to (143) wherein the color filters used inthe color filter arrays are identical among the modulating layers.(146) The system according to (143) wherein the color filters used inthe color filter arrays of each layer of the multi-layer display have afull-width half-max response of 70 nm or more.(147) The system according to (143) wherein the color filters used inthe color filter arrays of each layer of the multi-layer display have afull-width half-max response of more than 100 nm.(148) The system according to (2) wherein color images are shown bymeans of attaching color filter arrays to the modulating elements ofonly one modulating layer.(149) The system according to (148) wherein the images to be shown aredecomposed into a luminance channel that is sampled at the same samplerate as the modulators comprising the display layers, and a chrominancechannel that is subsampled below the rate of said modulators.(150) The system according to (2) wherein the reflected light pathbetween the display layers is included in the parameter space of thedisplay.(151) The system according to (150) wherein the displayed multi-viewimages are interpreted by a human viewer to represent one or morevirtual objects that extend over a depth range in a region in front ofthe display, behind the display, or both; and wherein the reflectedlight path between layers is exploited to extend said depth range to adistance of at least 1.5 times the spacing between the layers.(152) A parametric multi-view display system, comprising three or moremodulating layers wherein diffusers are placed between the displaylayers; and wherein said diffusers have weights that determine a degreeof optical blur; and wherein said diffuser weights are selected suchthat the degree of optical blur reduces the observed moiré interference.(153) The system according to (2) wherein the optimization result isconstrained to take on binary values (fully opaque or fullytransparent).(154) The system according to (153) wherein the modulating layers aremade from LCD panels; and wherein the LCD panels are binary Pi cells.(155) The system according to (153) wherein the modulating layers aremade from LCD panels; and wherein the LCD panels are binaryferroelectric liquid crystal arrays.(156) The system according to (153) wherein the modulating layers aremade from MEMS shutter devices.(157) The system according to (153) wherein the modulating layers aremade from MEMS mirror devices with accompanying optical elements.(158) The system according to (2) wherein the modulating elements of themodulating layers have fewer than 8-bits of precision (low bit depth);and wherein the parameter space is quantized to the precision of thedisplay devices.(159) The system according to (158) wherein the modulating elements ofthe modulating layers have fewer than 8-bits of precision (low bitdepth); and wherein at least one such low bit depth modulating layer isan LCD panel.(160) The system according to (2) wherein the contrast of at least oneof the modulating layers is less than 1:100.(161) The system according to (160) wherein the layer with contrastbelow 1:100 is made from an LCD.(162) The system according to (2) wherein at least one layer comprisesan array of light emitting diodes; and wherein at least one layercomprises a light attenuating modulator; and wherein said layerscomprising an array of light emitting diodes is placed behind at leastone of said layers comprising a light attenuating modulator, from theperspective of a viewer.(163) The system according to (2) wherein at least one of the modulatinglayers comprises a passive, patterned light attenuating material.(164) The system according to (163) wherein said passive materialcomprises a transparent plastic sheet, patterned with ink.(165) The system according to (164) wherein the ink is deposited by aninkjet printer(166) The system according to (163) wherein said passive materialcomprises a transparent plastic sheet, patterned by toner, e.g. from alaser printer or other electrostatic printing method.(167) The system according to (163) wherein said passive materialcomprises a glass sheet patterned by ink.(168) The system according to (163) wherein said passive materialcomprises an optically-exposed film, e.g. a photomask.(169) The system according to (163) wherein said passive material hasbeen patterned using a chromogenic process.(170) The system according to (2) wherein at least one of the modulatinglayers comprises a passive, machined layer.(171) The system according to (170) wherein the machined layer is madefrom wood.(172) The system according to (170) wherein the machined layer is madefrom metal.(173) The system according to (170) wherein the machined layer is madefrom opaque plastic.(174) The system according to (165) wherein the spacing between themodulating layers is less than 5 mm.(175) The system according to (165) wherein the spacing between themodulating layers is less than 60 times the width of the smallestfeature size of the modulating layer.(176) A parametric multi-view display system, comprising at least onemodulating layer, and at least one reflective layer.(177) The system according to (176) wherein the position of an incidentlight source is known by the system; and wherein the position of theviewer is known by the system.(178) The system according to (177) wherein the reflective layer has adiffuse lobe.(179) The system according to (177) wherein the reflective layer is alsoa modulating layer.(180) The system according to (177) wherein the reflective layercomprises an e-ink or e-paper display.(181) The system according to (177) wherein the reflective layer is alsotransmissive.(182) The system according to (181) wherein the reflective andtransmissie layer comprises a transflective LCD.(183) The system according to (2) wherein the display is illuminated bya backlight unit; and wherein the backlight unit emits more than 90% ofthe total light emitted by said backlight unit over the angular regioncontaining the expected viewing locations of the display.(184) The system according to (2) wherein the system provides a 3D dashdisplay in a vehicle.(185) The system according to (2) wherein the system provides a 3Dinstrument cluster in a vehicle.(186) The system according to (2) wherein the system provides a 3Dcontrol surface in a vehicle.(187) The system according to (2) wherein the system provides a 3D userinterface on a smartphone.(188) The system according to (2) wherein the system provides a 3D userinterface on a tablet computer.(189) The system according to (2) wherein the system provides a 3D userinterface on a laptop computer.(190) The system according to (2) wherein the system provides a 3D userinterface on a desktop computer.(191) The system according to (2) wherein the system provides a 3D userinterface on a workstation computer.(192) The system according to (2) wherein the system provides a 3Dvisualization on a CAD workstation.(193) The system according to (2) wherein the system provides a 3D viewof an architectural model.(194) The system according to (2) wherein the system provides a 3Dscreen for watching stereoscopic video content.(195) The system according to (2) wherein the system provides a 3Dscreen for watching multi-view television or other multi-view videocontent.(196) The system according to (195) wherein the display is in a privatehome.(197) The system according to (195) wherein the display is in a movietheater.(198) The system according to (2) wherein the system provides a 3Dgaming experience.(199) The system according to (198) wherein the 3D gaming experience ison a mobile device.(200) The system according to (198) wherein the 3D gaming experience isin a private home.(201) The system according to (198) wherein the 3D gaming experience ison a desktop computer.(202) The system according to (198) wherein the 3D gaming experience ison a console gaming system.(203) The system according to (198) wherein the 3D gaming experience isin a public arcade.(204) The system according to (2) wherein the system provides a 3D viewof medical imaging data.(205) The system according to (2) wherein the system provides a 3D viewof volumetric medical scans.(206) The system according to (2) wherein the system provides a 3D viewmedical chart data or vital patient data.(207) The system according to (2) wherein the system provides a 3D viewof seismic imaging data.(208) The system according to (2) wherein the system provides a 3D viewof microscopic structures.(209) The system according to (2) wherein the system provides a 3D viewof trends in large data sets.(210) The system according to (2) wherein the system provides a 3D viewof financial data.(211) The system according to (2) wherein the system provides a 3Dmedium for artistic works.(212) The system according to (2) wherein the system provides a 3Dadvertisement.(213) The system according to (2) wherein the system provides a 3Dannouncement.(214) The system according to (2) wherein the system provides a 3Dvisualization for remote vehicle operation.(215) The system according to (214) wherein the vehicle is airborne.(216) The system according to (214) wherein the vehicle is submersible.(217) The system according to (214) wherein the vehicle travels overland.(218) The system according to (214) wherein the vehicle operates inspace.(219) The system according to (214) wherein the vehicle works onextraplanetary bodies.(220) The system according to (2) wherein the system provides a 3Dvisualization for controlling a fleet of autonomous vehicles.(221) The system according to (2) wherein the system provides 3Dteleconferencing.(222) The system according to (2) wherein the system provides 3Dtelepresence.(223) The system according to (2) wherein the system provides 3Dteleoperation.(224) The system according to (163) wherein the system provides a 3Dgraphic or text functioning as an indicator light.(225) The system according to (163) wherein the system provides a 3Dgraphic or text functioning as an emblem in a control panel for anelectronic system.(226) The system according to (163) wherein the system provides a 3Dgraphic or text functioning as a label for or on a switch, button,capacitive-touch button, knob, slide control, optical proximity sensoror other physical control element or surface.(227) The system according to (163) wherein the system provides a 3Dgraphic or text overlaid on an external light in a vehicle, e.g. a brakelight, tail light or other external illuminated surface.(228) The system according to (163) wherein the system provides a 3Dadvertisement.(229) The system according to (163) wherein the system provides a 3Dannouncement.(230) The system according to (163) wherein layers in the system areadhered to glass, e.g. a window.(231) The system according to (2) wherein the displayed multi-viewimages are interpreted by a human viewer to represent one or morevirtual objects; and wherein the said virtual objects appear to coexistwith true objects in the physical world as perceived by a human viewer;and wherein the virtual objects appear to have 3D shape.(232) The system according to (231) wherein the light emitted from thesystem is viewed by a viewer through a semi-reflective screen.(233) The system according to (231) wherein said semi-reflective screenis the windshield of a car.(234) The system according to (231) wherein said semi-reflective screenis the cockpit of an aircraft.(235) The system according to (231) wherein said semi-reflective screenis the window of a building or structure.(236) The system according to (231) wherein the light emitted from themodulating layers is transmitted through a lens.(237) The system according to (231) wherein the light emitted from themodulating layers is transmitted through a catadioptric system.(238) The system according to (231) wherein the light emitted from themodulating layers is transmitted through a system of curved mirrors.(239) A system for optimizing actuation signals for use in an activeoptimized light field display, wherein a band-limited perceptual modelis utilized in performing the optimization.(240) A system for optimizing actuation signals for use in an activeoptimized light field display, wherein a band-limited perceptual modelis utilized in performing the optimization, and wherein the associatedoptical stack contains two or more active multiplicative layers, eachcontaining four or more individually-selectable pixels.(241) A system for optimizing actuation signals for use in an activeoptimized light field display, wherein a band-limited perceptual modelis utilized in performing the optimization, and wherein the associatedoptical stack contains two or more active multiplicative layers, eachcontaining four or more individually-selectable pixels or preconfiguredattenuation patterns.(242) A system for optimizing actuation signals for use in a passiveoptimized light field display, wherein a band-limited perceptual modelis utilized in performing the optimization.(243) A system for optimizing actuation signals for use in a light fieldemitter used in curing a photosensitive resin in a three-dimensionalprinter, wherein a band-limited model of the material is utilized.(244) A system for optimizing actuation signals for use in a light fieldemitter used in exposing a two-dimensional photographic medium, whereina band-limited model of the medium is utilized.(245) A system for optimizing actuation signals for use in a light fieldemitter used in exposing a biological tissue medium, wherein aband-limited model of the medium is utilized.(246) A method for optimizing actuation signals where a band-limitedperceptual model is utilized, wherein the associated computation isdistributed across multiple computational threads or resources.(247) A method for optimizing actuation signals where a band-limitedperceptual model is utilized, wherein the associated computation isdistributed across multiple computational threads or resources, andwherein each resource contributes in whole or in part to theoptimization of actuation signals for a corresponding layer in theoptical stack.(248) A method for optimizing actuation signals where a band-limitedperceptual model is utilized, wherein the distribution of computationdescribed according to (247) is further distributed across multiplecomputational threads or resources, and wherein each further distributedresource contributes in whole or in part to the optimization of acorresponding perceived view of the display.(249) A method for optimizing actuation signals where a band-limitedperceptual model is utilized, wherein the optimization method updatesthe actuation signals iteratively, and wherein the updates for eachiteration utilize a rule where previous values are multiplied by adynamically-generated step to obtain values in the subsequent iteration.(250) A system for caching, compositing and recalling optimizedactuation signals for use in an optimized light field display, whereinthe actuation signals may be stored, recalled and combined with otheractuation signals to obtain those actuation signals that are controllingthe display.(251) A method for compositing actuation signals in a light-fielddisplay wherein gradual edge blurring is performed between theboundaries of the source signals.(252) A method for compositing actuation signals in a light-fielddisplay wherein those portions of the source signals being selecteddepend on the location of the viewer.(253) A method for compositing actuation signals in a multiplicative,multi-layer light-field display wherein one or more of the sourcesignals are combined by modifying a single layer of the display.(254) A method for compositing actuation signals in a multiplicative,multi-layer light-field display wherein one or more of the sourcesignals are combined using an optimization algorithm.(255) A method for performing calibration of an optimized light fielddisplay wherein a mobile device is utilized in performing thecalibration.(256) A method for performing calibration of an optimized light fielddisplay wherein a mobile device is utilized in performing thecalibration, specifically utilizing an integrated camera, depth camera,or camera array in the mobile device in performing the calibration.(257) A method for performing calibration of an optimized light fielddisplay wherein a mobile device is utilized in performing thecalibration, specifically utilizing an accelerometer, gyroscope, compassor other related sensor in performing the calibration.(258) A method for performing calibration of an optimized light fielddisplay wherein a mobile device is utilized in performing thecalibration, and wherein a calibration pattern is displayed theoptimized light field display being calibrated.(259) A method for performing calibration of an optimized light fielddisplay wherein a mobile device is utilized in performing thecalibration, and wherein a calibration pattern is displayed the mobiledevice used in calibration.(260) A method for performing calibration of an optimized light fielddisplay in a manufacturing environment, using any such method that wouldfollow by adaptation of the methods specified according to (255)-(259)for use on industrial equipment by a person having ordinary skill in theart.

Having thus described several aspects some embodiments, it is to beappreciated that various alterations, modifications, and improvementswill readily occur to those skilled in the art. Such alterations,modifications, and improvements are intended to be within the spirit andscope of the present disclosure. Accordingly, the foregoing descriptionand drawings are by way of example only.

The above-described embodiments of the present disclosure can beimplemented in any of numerous ways. For example, the embodiments may beimplemented using hardware, software or a combination thereof. Whenimplemented in software, the software code can be executed on anysuitable processor or collection of processors, whether provided in asingle computer or distributed among multiple computers.

Also, the various methods or processes outlined herein may be coded assoftware that is executable on one or more processors that employ anyone of a variety of operating systems or platforms. Additionally, suchsoftware may be written using any of a number of suitable programminglanguages and/or programming or scripting tools, and also may becompiled as executable machine language code or intermediate code thatis executed on a framework or virtual machine.

In this respect, the concepts disclosed herein may be embodied as anon-transitory computer-readable medium (or multiple computer-readablemedia) (e.g., a computer memory, one or more floppy discs, compactdiscs, optical discs, magnetic tapes, flash memories, circuitconfigurations in Field Programmable Gate Arrays or other semiconductordevices, or other non-transitory, tangible computer storage medium)encoded with one or more programs that, when executed on one or morecomputers or other processors, perform methods that implement thevarious embodiments of the present disclosure discussed above. Thecomputer-readable medium or media can be transportable, such that theprogram or programs stored thereon can be loaded onto one or moredifferent computers or other processors to implement various aspects ofthe present disclosure as discussed above.

The terms “program” or “software” are used herein to refer to any typeof computer code or set of computer-executable instructions that can beemployed to program a computer or other processor to implement variousaspects of the present disclosure as discussed above. Additionally, itshould be appreciated that according to one aspect of this embodiment,one or more computer programs that when executed perform methods of thepresent disclosure need not reside on a single computer or processor,but may be distributed in a modular fashion amongst a number ofdifferent computers or processors to implement various aspects of thepresent disclosure.

Computer-executable instructions may be in many forms, such as programmodules, executed by one or more computers or other devices. Generally,program modules include routines, programs, objects, components, datastructures, etc. that perform particular tasks or implement particularabstract data types. The functionality of the program modules may becombined or distributed as desired in various embodiments.

Also, data structures may be stored in computer-readable media in anysuitable form. For simplicity of illustration, data structures may beshown to have fields that are related through location in the datastructure. Such relationships may likewise be achieved by assigningstorage for the fields with locations in a computer-readable medium thatconveys relationship between the fields. However, any suitable mechanismmay be used to establish a relationship between information in fields ofa data structure, including through the use of pointers, tags or othermechanisms that establish relationship between data elements.

Various features and aspects of the present disclosure may be usedalone, in any combination of two or more, or in a variety ofarrangements not specifically discussed in the embodiments described inthe foregoing and is therefore not limited in its application to thedetails and arrangement of components set forth in the foregoingdescription or illustrated in the drawings. For example, aspectsdescribed in one embodiment may be combined in any manner with aspectsdescribed in other embodiments.

Also, the concepts disclosed herein may be embodied as a method, ofwhich an example has been provided. The acts performed as part of themethod may be ordered in any suitable way. Accordingly, embodiments maybe constructed in which acts are performed in an order different thanillustrated, which may include performing some acts simultaneously, eventhough shown as sequential acts in illustrative embodiments.

Use of ordinal terms such as “first,” “second,” “third,” etc., in theclaims to modify a claim element does not by itself connote anypriority, precedence, or order of one claim element over another or thetemporal order in which acts of a method are performed, but are usedmerely as labels to distinguish one claim element having a certain namefrom another element having a same name (but for use of the ordinalterm) to distinguish the claim elements.

Also, the phraseology and terminology used herein is for the purpose ofdescription and should not be regarded as limiting. The use of“including,” “comprising,” “having,” “containing,” “involving,” andvariations thereof herein, is meant to encompass the items listedthereafter and equivalents thereof as well as additional items.

What is claimed is:
 1. A method of manufacturing a light field print,the light field print comprising a front transparent layer and a backtransparent layer, the method comprising: obtaining a first targetpattern for the front transparent layer, wherein the first targetpattern for the front transparent layer was generated using both contentto be displayed using the light field print and printing processinformation; obtaining a second target pattern for the back transparentlayer, wherein the second target pattern for the back transparent layerwas generated using both the content to be displayed using the lightfield print and the printing process information; printing the firsttarget pattern on the front transparent layer by depositing printingmaterial on the front transparent layer in accordance with the firsttarget pattern; and printing the second target pattern on the backtransparent layer by depositing printing material on the backtransparent layer in accordance with the second target pattern, whereinthe front transparent layer is spaced in depth at a distance from theback transparent layer, which distance is less than or equal to sixmillimeters and/or L/60, whichever is greater, wherein L is a maximumlinear extent of a larger one of the front transparent layer and theback transparent layer, when the front transparent layer and the backtransparent layer are different sizes, and a maximum linear extent ofthe front transparent layer when the front transparent layer and theback transparent layer are a same size.
 2. The method of claim 1,further comprising: assembling the light field print from the fronttransparent layer and the back transparent layer.
 3. The method of claim2, wherein the assembling comprises: after printing the second targetpattern on the back transparent layer and before printing the firsttarget pattern on the front transparent layer, placing the fronttransparent layer on the back transparent layer; and after placing thefront transparent layer on the back transparent layer, printing thefirst target pattern on the front transparent layer.
 4. The method ofclaim 1, wherein depositing printing material on the front transparentlayer comprises depositing ink or toner on the front transparent layer.5. The method of claim 1, comprising printing the first target patternwith a dot pitch of less than or equal to 0.0025 inches.
 6. The methodof claim 1, wherein the front transparent layer and the back transparentlayer are two sides of a same print medium, and wherein printing thefirst target pattern on the front transparent layer and printing thesecond target on the back transparent layer comprises: printing thefirst target pattern on a first side of the print medium; and printingthe second target pattern on a second side of the print medium.
 7. Themethod of claim 1, wherein the front transparent layer and the backtransparent layer are on different substrates.
 8. The method of claim 1,wherein printing the first and second target patterns is performed by adigital printing system.
 9. The method of claim 1, wherein the printingprocess information comprises information characterizing how much dotgain results from the printing process, information indicating a maximumallowable ink density of the printing medium, and/or informationindicating a dot pitch of the prints generated by the printing process.10. A method of manufacturing a light field print, the methodcomprising: obtaining content to be displayed using a light field print;obtaining printing process information including information about dotgain of the printing process; generating, based at least in part on thecontent and the printing process information, a first target pattern anda second target pattern; printing the first target pattern on a firstside of a print medium by depositing printing material on the first sideof the print medium in accordance with the first target pattern; andprinting the second target pattern on a second side of the print mediumby depositing printing material on the second side of the print mediumin accordance with the second target pattern.
 11. The method of claim10, wherein depositing printing material on the front transparent layercomprises depositing ink or toner on the front transparent layer. 12.The method of claim 10, comprising printing the first target patternwith a dot pitch of less than or equal to 0.0025 inches.
 13. The methodof claim 10, wherein printing the first and second target patterns isperformed by a digital printing system.
 14. The method of claim 10,wherein the printing process information comprises informationcharacterizing how much dot gain results from the printing process,information indicating a maximum allowable ink density of the printingmedium, and/or information indicating a dot pitch of the printsgenerated by the printing process.
 15. A method of manufacturing a lightfield print, the method comprising: obtaining a first target patterngenerated based at least in part on content to be displayed using thelight field print and printing process information including informationabout dot gain of the printing process; obtaining a second targetpattern generated based at least in part on the content to be displayedusing the light field print and the printing process informationincluding information about the dot gain of the printing process;printing the first target pattern on a first side of a print medium bydepositing printing material on the first side of the print medium inaccordance with the first target pattern; and printing the second targetpattern on a second side of the print medium by depositing printingmaterial on the second side of the print medium in accordance with thesecond target pattern.
 16. The method of claim 15, wherein depositingprinting material on the front transparent layer comprises depositingink or toner on the front transparent layer.
 17. The method of claim 15,comprising printing the first target pattern with a dot pitch of lessthan or equal to 0.0025 inches.
 18. The method of claim 15, whereinprinting the first and second target patterns is performed by a digitaloffset printing system.
 19. The method of claim 15, wherein the printingprocess information comprises information characterizing how much dotgain results from the printing process, information indicating a maximumallowable ink density of the printing medium, and/or informationindicating a dot pitch of the prints generated by the printing process.20. The method of claim 15, further comprising generating the firsttarget pattern based at least in part on the content to be displayedusing the light field print and the printing process informationincluding information about the dot gain of the printing process.